diff --git a/.gitignore b/.gitignore index 3540bb2..0d80b28 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,7 @@ Spring-Boot/target Spring-Netty/target Spring-Security/target rocketmqdemo/target +architecture/target # .DO_Store .DS_Store diff --git a/README.md b/README.md index 7ca38a0..5f8a663 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,8 @@ -![JavaSourceCodeLearningImage](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/images/JavaSourceCodeLearningImage.png) +

Java Source Code Learning

+ +

+ 一份面向 Java 后端工程师的源码阅读地图:从 JDK / JUC 到 Spring、Netty、Kafka、RocketMQ,按核心链路拆解框架设计与底层实现。 +

@@ -20,132 +24,179 @@ Netty - rocketmq + RocketMQ - rocketmq + Kafka

-Java相关流行框架源码分析,学习以及总结,项目持续更新中。 - -框架或者源码包括: - -✅ JDK源码 - -✅ JUC源码 - -✅ Spring源码 - -✅ SpringBoot源码 - -✅ SpringAOP源码 - -✅ SpringSecurity源码 - -✅ SpringSecurity OAuth2源码 - -✅ JDK源码 - -✅ Dubbo源码 - -✅ Netty源码 - -✅ RocketMQ源码 - -✅ kafka源码 - -> 为什么要分析、学习源码? - - 学习框架源码不仅能帮助我们在实际问题出现时快速定位问题、理解根因并高效解决,还能深入掌握框架的整体架构设计思路与核心设计模式,从而提升自身的系统设计能力与架构思维。 -同时,通过学习优秀开源框架的底层实现,可以不断强化对复杂系统拆分、模块协作以及性能优化的理解,这对于个人技术能力的长期成长至关重要。因此,源码学习虽然过程相对枯燥,但这是提升架构设计能力的必经路径。只有持续积累,才能在实际系统设计与工程实践中做到真正的“游刃有余”。 - -# 目录 - -- kafka源码分析 - - kafka版本:4.2 - - [kafka 核心概念扫描](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/Kafka%E6%A0%B8%E5%BF%83%E6%A6%82%E5%BF%B5%E6%89%AB%E7%9B%B2.md) - - [kafka broker核心源码分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20broker%E6%A0%B8%E5%BF%83%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90.md) - - [kafka broker核心源码分析——生产者篇](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20broker%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%E2%80%94%E2%80%94%E7%94%9F%E4%BA%A7%E8%80%85%E7%AF%87.md) - - [kafka消费者核心源码分析(一)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%E6%B6%88%E8%B4%B9%E8%80%85%E6%A0%B8%E5%BF%83%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%80%EF%BC%89.md) - - [kafka Rebalance核心逻辑分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20rebalance%E6%A0%B8%E5%BF%83%E9%80%BB%E8%BE%91%E5%88%86%E6%9E%90.md) - - [kafka ISR原理](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/Kafka%20ISR%20%E5%BA%95%E5%B1%82%E5%8E%9F%E7%90%86.md) - -- JDK源码学习 - - JDK版本:1.8.0_77 - - [深入学习String源码与底层(一)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0String%E6%BA%90%E7%A0%81%E4%B8%8E%E5%BA%95%E5%B1%82%EF%BC%88%E4%B8%80%EF%BC%89.md) - - [深入学习String源码与底层(二)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0String%E6%BA%90%E7%A0%81%E4%B8%8E%E5%BA%95%E5%B1%82%EF%BC%88%E4%BA%8C%EF%BC%89.md) - - [深入解读CompletableFuture源码与原理](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E8%A7%A3%E8%AF%BBCompletableFuture%E6%BA%90%E7%A0%81%E4%B8%8E%E5%8E%9F%E7%90%86.md) - - [深入分析ThreadLocal](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%88%86%E6%9E%90ThreadLocal.md) - - [深入学习Java volatile关键字](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0Java%20volatile%E5%85%B3%E9%94%AE%E5%AD%97.md) - - [深入学习Thread底层原理](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0Thread%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81.md) - - [深入学习JDK1.7、8 HashMap扩容原理]() - - [开源项目里那些看不懂的位运算分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E9%87%8C%E9%82%A3%E4%BA%9B%E7%9C%8B%E4%B8%8D%E6%87%82%E7%9A%84%E4%BD%8D%E8%BF%90%E7%AE%97%E5%88%86%E6%9E%90.md) - - [ThreadPoolExecutor源码分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E8%A7%A3%E6%9E%90ThreadPoolExecutor%E5%BA%95%E5%B1%82%E5%8E%9F%E7%90%86.md) - -- Spring源码学习 - - Spring版本:5.2.1.RELEASE - - - [深入Spring源码系列(一)——在IDEA中构建Spring源码](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%E5%9C%A8IDEA%E4%B8%AD%E6%9E%84%E5%BB%BASpring%E6%BA%90%E7%A0%81.md) - - [深入Spring源码系列(二)——深入Spring容器,通过源码阅读和时序图来彻底弄懂Spring容器(上)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%E6%B7%B1%E5%85%A5Spring%E5%AE%B9%E5%99%A8%EF%BC%8C%E9%80%9A%E8%BF%87%E6%BA%90%E7%A0%81%E9%98%85%E8%AF%BB%E5%92%8C%E6%97%B6%E5%BA%8F%E5%9B%BE%E6%9D%A5%E5%BD%BB%E5%BA%95%E5%BC%84%E6%87%82Spring%E5%AE%B9%E5%99%A8%EF%BC%88%E4%B8%8A%EF%BC%89.md) - - [深入Spring源码系列(二)——深入Spring容器,通过源码阅读和时序图来彻底弄懂Spring容器(下)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%E6%B7%B1%E5%85%A5Spring%E5%AE%B9%E5%99%A8%EF%BC%8C%E9%80%9A%E8%BF%87%E6%BA%90%E7%A0%81%E9%98%85%E8%AF%BB%E5%92%8C%E6%97%B6%E5%BA%8F%E5%9B%BE%E6%9D%A5%E5%BD%BB%E5%BA%95%E5%BC%84%E6%87%82Spring%E5%AE%B9%E5%99%A8%EF%BC%88%E4%B8%8B%EF%BC%89.md) - - [深入Spring源码系列(补充篇)——程序调用Spring源码](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E8%A1%A5%E5%85%85%E7%AF%87%EF%BC%89%E2%80%94%E2%80%94%E7%A8%8B%E5%BA%8F%E8%B0%83%E7%94%A8Spring%E6%BA%90%E7%A0%81.md) - - [从Spring源码中学习——策略模式](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E4%BB%8ESpring%E6%BA%90%E7%A0%81%E4%B8%AD%E5%AD%A6%E4%B9%A0%E2%80%94%E2%80%94%E7%AD%96%E7%95%A5%E6%A8%A1%E5%BC%8F.md) - -- SpringAOP源码学习 - - Spring版本:5.2.1.RELEASE - - - [深入学习SpringAOP源码(一)——注册AnnotationAwareAspectJAutoProxyCreator](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%E6%B3%A8%E5%86%8CAnnotationAwareAspectJAutoProxyCreator.md) - - [深入学习SpringAOP源码(二)—— 深入AnnotationAwareAspectJAutoProxyCreator](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%20%E6%B7%B1%E5%85%A5AnnotationAwareAspectJAutoProxyCreator.md) - - [深入学习SpringAOP源码(三)——揭开JDK动态代理和CGLIB代理的神秘面纱](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%B8%89%EF%BC%89%E2%80%94%E2%80%94%E6%8F%AD%E5%BC%80JDK%E5%8A%A8%E6%80%81%E4%BB%A3%E7%90%86%E5%92%8CCGLIB%E4%BB%A3%E7%90%86%E7%9A%84%E7%A5%9E%E7%A7%98%E9%9D%A2%E7%BA%B1.md) - -- SpringBoot源码学习 - - SpringBoot版本:2.2.1.RELEASE - - - [深入浅出SpringBoot源码——SpringFactoriesLoader](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94SpringFactoriesLoader.md) - - [深入浅出SpringBoot源码——监听器与事件机制](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E7%9B%91%E5%90%AC%E5%99%A8%E4%B8%8E%E4%BA%8B%E4%BB%B6%E6%9C%BA%E5%88%B6.md) - - [深入浅出SpringBoot源码——系统初始化器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E7%B3%BB%E7%BB%9F%E5%88%9D%E5%A7%8B%E5%8C%96%E5%99%A8.md) - - [深入浅出SpringBoot源码——启动加载器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E5%90%AF%E5%8A%A8%E5%8A%A0%E8%BD%BD%E5%99%A8.md) - -- SpringSecurity&OAuth2源码学习 - - SpringSecurity版本:5.1.0.RELEASE - - [深入浅出SpringSecurity和OAuth2(一)—— 初识SpringSecurity](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%20%E5%88%9D%E8%AF%86SpringSecurity.md) - - [深入浅出SpringSecurity和OAuth2(二)—— 安全过滤器FilterChainProxy](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%20%E5%AE%89%E5%85%A8%E8%BF%87%E6%BB%A4%E5%99%A8FilterChainProxy.md) - - [深入浅出SpringSecurity和OAuth2(三)—— WebSecurity建造核心逻辑](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%B8%89%EF%BC%89%E2%80%94%E2%80%94%20WebSecurity%E5%BB%BA%E9%80%A0%E6%A0%B8%E5%BF%83%E9%80%BB%E8%BE%91.md) - - [深入浅出SpringSecurity和OAuth2(四)—— FilterChainProxy过滤器链中的几个重要的过滤器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E5%9B%9B%EF%BC%89%E2%80%94%E2%80%94%20FilterChainProxy%E8%BF%87%E6%BB%A4%E5%99%A8%E9%93%BE%E4%B8%AD%E7%9A%84%E5%87%A0%E4%B8%AA%E9%87%8D%E8%A6%81%E7%9A%84%E8%BF%87%E6%BB%A4%E5%99%A8.md) - -- Netty底层源码解析 - - Netty版本:4.1.43.Final - - [Netty概念扫盲](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E6%A6%82%E5%BF%B5%E6%89%AB%E7%9B%B2.md) - - [二进制运算以及源码、反码以及补码学习](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/%E4%BA%8C%E8%BF%9B%E5%88%B6.md) - - [Netty源码包结构](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/Netty%E6%BA%90%E7%A0%81%E5%8C%85%E7%BB%93%E6%9E%84.md) - - [Netty底层源码解析-EventLoopGroup](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/Netty%E4%B8%AD%E7%9A%84EventLoopGroup%E6%98%AF%E4%BB%80%E4%B9%88.md) - - [Netty底层源码解析-初始Netty及其架构](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-%E5%88%9D%E5%A7%8BNetty%E5%8F%8A%E5%85%B6%E6%9E%B6%E6%9E%84.md) - - [Netty底层源码解析-Netty服务端启动分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-Netty%E6%9C%8D%E5%8A%A1%E7%AB%AF%E5%90%AF%E5%8A%A8%E5%88%86%E6%9E%90.md) - - [Netty底层源码解析-NioEventLoop原理分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-NioEventLoop%E5%8E%9F%E7%90%86%E5%88%86%E6%9E%90.md) - - [Netty底层源码解析-ChannelPipeline分析(上)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-ChannelPipeline%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%8A%EF%BC%89.md) - - [Netty底层源码解析-ChannelPipeline分析(下)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-ChannelPipeline%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%8B%EF%BC%89.md) - - [Netty底层源码解析-NioServerSocketChannel接受数据原理分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-NioServerSocketChannel%E6%8E%A5%E5%8F%97%E6%95%B0%E6%8D%AE%E5%8E%9F%E7%90%86%E5%88%86%E6%9E%90.md) - - Netty底层源码解析-NioSocketChannel接受、发送数据原理分析 - - Netty底层源码解析-FastThreadLocal原理分析 - - Netty底层源码解析-内存分配原理分析 - - Netty底层源码解析-RocketMQ底层使用到的Netty - -Netty实战课相关点位于:Spring-Netty,com/bruis/learnnetty/im包下,有需要的读者可前往查看。 - - -- RocketMQ底层源码解析 - - RocketMQ版本:4.9.0 - - RocketMQ底层源码解析-RocketMQ环境搭建 - - RocketMQ底层源码解析-本地调试RocketMQ源码 - - RocketMQ底层源码解析-NameServer分析 - - 持续更新中... - - - -# 支持 - - 原创不易,各位帅哥美女star支持下... +## 项目亮点 + +| 你能看到什么 | 重点能力 | +| --- |--------------------------------------------| +| JDK / JUC 源码 | 集合、并发、线程池、内存模型、CompletableFuture | +| Spring / SpringBoot 源码 | IOC 容器、事件机制、启动流程、自动装配、扩展点 | +| SpringAOP / Security / OAuth2 | 代理机制、过滤器链、安全认证授权主流程 | +| Netty 源码 | Reactor 模型、EventLoop、ChannelPipeline、网络通信链路 | +| Kafka / RocketMQ 源码 | Broker、Producer、Consumer、Rebalance、消息存储与复制 | + +## 学习路线 + +```text +JDK / JUC 基础源码 + ↓ +Spring 容器与扩展点 + ↓ +SpringBoot 启动与自动装配 + ↓ +Netty 网络通信模型 + ↓ +Kafka / RocketMQ 消息系统源码 +``` + +## 为什么要读源码? + +源码学习不是为了记住每一行实现,而是为了把框架背后的设计选择看清楚。 + +当线上问题出现时,读过核心链路的人通常能更快定位边界、判断根因、验证假设;当自己做系统设计时,也更容易借鉴成熟框架在模块拆分、扩展点设计、并发控制、性能优化上的经验。 + +这个项目会围绕主流 Java 后端框架的核心路径持续整理源码分析、学习笔记和关键图解。 + +## 内容目录 + +
+Kafka 源码分析 + +- Kafka 版本:4.3 +- [Kafka 核心概念扫盲](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/Kafka%E6%A0%B8%E5%BF%83%E6%A6%82%E5%BF%B5%E6%89%AB%E7%9B%B2.md) +- [Kafka Broker 核心源码分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20broker%E6%A0%B8%E5%BF%83%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90.md) +- [Kafka Broker 源码分析:生产者篇](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20broker%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%E2%80%94%E2%80%94%E7%94%9F%E4%BA%A7%E8%80%85%E7%AF%87.md) +- [Kafka 消费者核心源码分析(一)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%E6%B6%88%E8%B4%B9%E8%80%85%E6%A0%B8%E5%BF%83%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%80%EF%BC%89.md) +- [Kafka Rebalance 核心逻辑分析(Classic Consumer Group Protocol)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/kafka%20Rebalance%E6%A0%B8%E5%BF%83%E9%80%BB%E8%BE%91%E5%88%86%E6%9E%90.md) +- [Kafka ISR 原理](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/kafka/Kafka%20ISR%20%E5%BA%95%E5%B1%82%E5%8E%9F%E7%90%86.md) + +Kafka架构图 +![kafka_architecture](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_architecture.png) + +Kafka Rebalance流程图(Classic Consumer Group Protocol 经典消费者组协议) +![Kafka EAGER Rebalance](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_EAGER_rebalance.png) + +![Kafka Cooperative Rebalance](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_COOPERATIVE_rebalance.png) + +Kafka ISR / HW / LEO关系图 +![Kafka_HW_ISR_LEO](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_HW_ISR_LEO.png) + + +
+ +
+JDK / JUC 源码学习 + +- JDK 版本:1.8.0_77 +- [深入学习 String 源码与底层(一)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0String%E6%BA%90%E7%A0%81%E4%B8%8E%E5%BA%95%E5%B1%82%EF%BC%88%E4%B8%80%EF%BC%89.md) +- [深入学习 String 源码与底层(二)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0String%E6%BA%90%E7%A0%81%E4%B8%8E%E5%BA%95%E5%B1%82%EF%BC%88%E4%BA%8C%EF%BC%89.md) +- [深入解读 CompletableFuture 源码与原理](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E8%A7%A3%E8%AF%BBCompletableFuture%E6%BA%90%E7%A0%81%E4%B8%8E%E5%8E%9F%E7%90%86.md) +- [深入分析 ThreadLocal](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%88%86%E6%9E%90ThreadLocal.md) +- [深入学习 Java volatile 关键字](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0Java%20volatile%E5%85%B3%E9%94%AE%E5%AD%97.md) +- [深入学习 Thread 底层原理](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0Thread%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81.md) +- [深入学习HashMap 底层源码与原理](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/HashMap%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90.md) +- [开源项目里那些看不懂的位运算分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E9%87%8C%E9%82%A3%E4%BA%9B%E7%9C%8B%E4%B8%8D%E6%87%82%E7%9A%84%E4%BD%8D%E8%BF%90%E7%AE%97%E5%88%86%E6%9E%90.md) +- [ThreadPoolExecutor 源码分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/%E6%B7%B1%E5%85%A5%E8%A7%A3%E6%9E%90ThreadPoolExecutor%E5%BA%95%E5%B1%82%E5%8E%9F%E7%90%86.md) +- [AQS 源码分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/JDK/AQS%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90.md) + +HashMap原理图 +![HashMap原理图](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/HashMap.png) + +AQS架构图 +![AQS架构图](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/AQS.png) + +
+ +
+Spring 源码学习 + +- Spring 版本:5.2.1.RELEASE +- [深入 Spring 源码系列(一):在 IDEA 中构建 Spring 源码](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%E5%9C%A8IDEA%E4%B8%AD%E6%9E%84%E5%BB%BASpring%E6%BA%90%E7%A0%81.md) +- [深入 Spring 容器源码与时序图(上)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%E6%B7%B1%E5%85%A5Spring%E5%AE%B9%E5%99%A8%EF%BC%8C%E9%80%9A%E8%BF%87%E6%BA%90%E7%A0%81%E9%98%85%E8%AF%BB%E5%92%8C%E6%97%B6%E5%BA%8F%E5%9B%BE%E6%9D%A5%E5%BD%BB%E5%BA%95%E5%BC%84%E6%87%82Spring%E5%AE%B9%E5%99%A8%EF%BC%88%E4%B8%8A%EF%BC%89.md) +- [深入 Spring 容器源码与时序图(下)](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%E6%B7%B1%E5%85%A5Spring%E5%AE%B9%E5%99%A8%EF%BC%8C%E9%80%9A%E8%BF%87%E6%BA%90%E7%A0%81%E9%98%85%E8%AF%BB%E5%92%8C%E6%97%B6%E5%BA%8F%E5%9B%BE%E6%9D%A5%E5%BD%BB%E5%BA%95%E5%BC%84%E6%87%82Spring%E5%AE%B9%E5%99%A8%EF%BC%88%E4%B8%8B%EF%BC%89.md) +- [深入 Spring 源码系列(补充篇):程序调用 Spring 源码](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E6%B7%B1%E5%85%A5Spring%E6%BA%90%E7%A0%81%E7%B3%BB%E5%88%97%EF%BC%88%E8%A1%A5%E5%85%85%E7%AF%87%EF%BC%89%E2%80%94%E2%80%94%E7%A8%8B%E5%BA%8F%E8%B0%83%E7%94%A8Spring%E6%BA%90%E7%A0%81.md) +- [从 Spring 源码中学习策略模式](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Spring/%E4%BB%8ESpring%E6%BA%90%E7%A0%81%E4%B8%AD%E5%AD%A6%E4%B9%A0%E2%80%94%E2%80%94%E7%AD%96%E7%95%A5%E6%A8%A1%E5%BC%8F.md) + +
+ +
+SpringAOP 源码学习 + +- Spring 版本:5.2.1.RELEASE +- [深入学习 SpringAOP 源码(一):注册 AnnotationAwareAspectJAutoProxyCreator](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%E6%B3%A8%E5%86%8CAnnotationAwareAspectJAutoProxyCreator.md) +- [深入学习 SpringAOP 源码(二):深入 AnnotationAwareAspectJAutoProxyCreator](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%20%E6%B7%B1%E5%85%A5AnnotationAwareAspectJAutoProxyCreator.md) +- [深入学习 SpringAOP 源码(三):揭开 JDK 动态代理和 CGLIB 代理的神秘面纱](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringAOP/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0SpringAOP%E6%BA%90%E7%A0%81%EF%BC%88%E4%B8%89%EF%BC%89%E2%80%94%E2%80%94%E6%8F%AD%E5%BC%80JDK%E5%8A%A8%E6%80%81%E4%BB%A3%E7%90%86%E5%92%8CCGLIB%E4%BB%A3%E7%90%86%E7%9A%84%E7%A5%9E%E7%A7%98%E9%9D%A2%E7%BA%B1.md) + +
+ +
+SpringBoot 源码学习 + +- SpringBoot 版本:2.2.1.RELEASE +- [深入浅出 SpringBoot 源码:SpringFactoriesLoader](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94SpringFactoriesLoader.md) +- [深入浅出 SpringBoot 源码:监听器与事件机制](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E7%9B%91%E5%90%AC%E5%99%A8%E4%B8%8E%E4%BA%8B%E4%BB%B6%E6%9C%BA%E5%88%B6.md) +- [深入浅出 SpringBoot 源码:系统初始化器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E7%B3%BB%E7%BB%9F%E5%88%9D%E5%A7%8B%E5%8C%96%E5%99%A8.md) +- [深入浅出 SpringBoot 源码:启动加载器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringBoot/%E6%B7%B1%E5%85%A5SpringBoot%E6%BA%90%E7%A0%81%E5%AD%A6%E4%B9%A0%E4%B9%8B%E2%80%94%E2%80%94%E5%90%AF%E5%8A%A8%E5%8A%A0%E8%BD%BD%E5%99%A8.md) + +
+ +
+SpringSecurity / OAuth2 源码学习 + +- SpringSecurity 版本:5.1.0.RELEASE +- [深入浅出 SpringSecurity 和 OAuth2(一):初识 SpringSecurity](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%B8%80%EF%BC%89%E2%80%94%E2%80%94%20%E5%88%9D%E8%AF%86SpringSecurity.md) +- [深入浅出 SpringSecurity 和 OAuth2(二):安全过滤器 FilterChainProxy](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%BA%8C%EF%BC%89%E2%80%94%E2%80%94%20%E5%AE%89%E5%85%A8%E8%BF%87%E6%BB%A4%E5%99%A8FilterChainProxy.md) +- [深入浅出 SpringSecurity 和 OAuth2(三):WebSecurity 建造核心逻辑](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E4%B8%89%EF%BC%89%E2%80%94%E2%80%94%20WebSecurity%E5%BB%BA%E9%80%A0%E6%A0%B8%E5%BF%83%E9%80%BB%E8%BE%91.md) +- [深入浅出 SpringSecurity 和 OAuth2(四):FilterChainProxy 过滤器链中的几个重要过滤器](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/SpringSecurity/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%B3%BB%E7%BB%9F%E5%AD%A6%E4%B9%A0SpringSecurity%E5%92%8COAuth2%EF%BC%88%E5%9B%9B%EF%BC%89%E2%80%94%E2%80%94%20FilterChainProxy%E8%BF%87%E6%BB%A4%E5%99%A8%E9%93%BE%E4%B8%AD%E7%9A%84%E5%87%A0%E4%B8%AA%E9%87%8D%E8%A6%81%E7%9A%84%E8%BF%87%E6%BB%A4%E5%99%A8.md) + +
+ +
+Netty 底层源码解析 + +- Netty 版本:4.2 +- [Netty 概念扫盲](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E6%A6%82%E5%BF%B5%E6%89%AB%E7%9B%B2.md) +- [二进制运算以及源码、反码以及补码学习](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/%E4%BA%8C%E8%BF%9B%E5%88%B6.md) +- [Netty 源码包结构](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/Netty%E6%BA%90%E7%A0%81%E5%8C%85%E7%BB%93%E6%9E%84.md) +- [Netty 底层源码解析:EventLoopGroup](https://github.com/coderbruis/JavaSourceLearning/blob/master/note/Netty/Netty%E4%B8%AD%E7%9A%84EventLoopGroup%E6%98%AF%E4%BB%80%E4%B9%88.md) +- [Netty 底层源码解析:初始 Netty 及其架构](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-%E5%88%9D%E5%A7%8BNetty%E5%8F%8A%E5%85%B6%E6%9E%B6%E6%9E%84.md) +- [Netty 底层源码解析:Netty 服务端启动分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-Netty%E6%9C%8D%E5%8A%A1%E7%AB%AF%E5%90%AF%E5%8A%A8%E5%88%86%E6%9E%90.md) +- [Netty 底层源码解析:NioEventLoop 原理分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-NioEventLoop%E5%8E%9F%E7%90%86%E5%88%86%E6%9E%90.md) +- [Netty 底层源码解析:ChannelPipeline 分析(上)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-ChannelPipeline%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%8A%EF%BC%89.md) +- [Netty 底层源码解析:ChannelPipeline 分析(下)](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-ChannelPipeline%E5%88%86%E6%9E%90%EF%BC%88%E4%B8%8B%EF%BC%89.md) +- [Netty 底层源码解析:NioServerSocketChannel 接受数据原理分析](https://github.com/coderbruis/JavaSourceCodeLearning/blob/master/note/Netty/Netty%E5%BA%95%E5%B1%82%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90-NioServerSocketChannel%E6%8E%A5%E5%8F%97%E6%95%B0%E6%8D%AE%E5%8E%9F%E7%90%86%E5%88%86%E6%9E%90.md) +- Netty 底层源码解析:NioSocketChannel 接受、发送数据原理分析 +- Netty 底层源码解析:FastThreadLocal 原理分析 +- Netty 底层源码解析:内存分配原理分析 +- Netty 底层源码解析:RocketMQ 底层使用到的 Netty + +Netty 实战课相关代码位于 `Spring-Netty` 模块下的 `com/bruis/learnnetty/im` 包。 + +Netty架构图 +![Netty整体架构图](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/Netty4.2.x.png) + +Netty主从Reactor架构图 +![Netty主从Reactor架构图](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/Netty4.2.x.Reactor.png) + +
+ +
+RocketMQ 底层源码解析 + +- RocketMQ 版本:4.9.0 +- RocketMQ 底层源码解析:RocketMQ 环境搭建 +- RocketMQ 底层源码解析:本地调试 RocketMQ 源码 +- RocketMQ 底层源码解析:NameServer 分析 +- 持续更新中... + +
+## 支持 +如果这个项目对你有帮助,欢迎 Star。源码学习是长期工程,我会持续补充更多核心链路分析、图解和实践示例。 diff --git a/architecture/README.md b/architecture/README.md new file mode 100644 index 0000000..bc3298a --- /dev/null +++ b/architecture/README.md @@ -0,0 +1,279 @@ +# 架构方案伪代码库 + +`architecture` module 用于沉淀不同业务场景下的架构思路、设计方案和关键伪代码。它关注的是 +复杂问题如何拆分、核心流程如何协作、数据一致性如何保证,以及不同技术组件之间的边界,而不是提供 +可以直接上线的完整业务系统。 + +当前实现以机票业务为背景,模拟以下两个相互关联的架构场景: + +1. **多供应商并发搜索**:一次机票查询并发分发给多个供应商,通过异步回调聚合结果,并支持超时返回部分结果。 +2. **海量政策高性能匹配**:使用 RocksDB 保存政策明细、RoaringBitmap 构建匹配索引,并通过版本化快照实现运行时无损切换。 + +后续会继续增加交易、下单等其他场景的架构伪代码,使该 module 逐步形成可复用的架构与设计方案集合。 + +## 模块目标 + +- 用最少的代码表达架构中的核心职责、协作关系和一致性约束。 +- 为类似业务问题提供可讨论、可演进的设计参考。 +- 通过接口隔离外部系统,使方案不被 Dubbo、Kafka、Redis 等具体技术绑定。 +- 通过场景说明和关键注释表达核心架构行为及约束。 +- 持续积累搜索、政策匹配、交易、下单等不同场景的方案。 + +本模块属于架构伪代码,生产落地时仍需根据实际情况补充鉴权、限流、熔断、监控、链路追踪、异常分级、 +数据迁移、容量评估、容灾和部署方案。 + +## 场景目录 + +| 场景 | 核心问题 | 当前状态 | +| --- | --- | --- | +| 多供应商并发搜索 | 任务分发、异步回调、结果聚合、超时与迟到回调 | 已实现伪代码 | +| 海量政策匹配 | 全量加载、增量追平、位图索引、快照校验与热切换 | 已实现伪代码 | +| 交易/下单 | 幂等下单、受控状态流转、乐观并发与失败隔离 | 已实现核心伪代码 | + +## 分层结构 + +代码先区分项目级公开契约、公共能力和具体业务场景,再在场景内部按照应用、领域和基础设施分层。 + +```text +com.arch.policy +├── api +│ ├── book # 下单及订单状态变更契约 +│ └── search # 项目级搜索契约、请求和响应 DTO +├── common +│ ├── config # Spring Bean 装配 +│ └── model # 可跨场景复用的模型 +├── search +│ ├── application # 查询编排、回调处理、启动和增量更新用例 +│ ├── domain +│ │ └── snapshot # RocksDB/位图快照领域逻辑 +│ └── infrastructure +│ ├── demo # 模拟供应商 +│ ├── kafka # 政策变更消息适配器 +│ ├── redis # 聚合状态和完成通知适配器 +│ └── rpc # Dubbo 接口实现 +├── book +│ ├── application # 幂等下单、状态变更用例和持久化端口 +│ ├── domain # 订单聚合、状态枚举和合法迁移规则 +│ └── infrastructure # 内存仓储示例和 Dubbo 适配器 +└── PolicySearchApplication # 当前搜索场景的启动入口 +``` + +依赖方向如下: + +```mermaid +flowchart LR + Caller["外部调用方"] --> API["api.search
项目级公开契约"] + Infrastructure["search.infrastructure
RPC / Redis / Kafka"] --> API + Infrastructure --> Application["search.application
流程编排"] + Application --> Domain["search.domain
领域规则"] + Application --> Common["common
公共配置与模型"] + Domain --> Common +``` + +公开 API 不依赖场景内部实现,领域层不依赖 Dubbo、Redis、Kafka 等外部技术。后续增加交易场景时, +可以平行新增 `api.order` 和 `order.application/domain/infrastructure`,避免交易代码与搜索代码混杂。 +只有真正跨场景稳定复用的模型或能力才应放入 `common`。 + +## 场景一:多供应商并发搜索 + +查询服务将一次请求分发给多个供应商。供应商可以分批回调结果,最后一次回调负责声明该供应商完成。 +Redis 保存聚合结果、待完成供应商集合和查询状态,是整个流程的唯一事实来源。 + +```mermaid +sequenceDiagram + autonumber + actor Caller as 调用方 + participant SearchAPI as PolicySearchRpcService + participant Coordinator as AsyncSearchCoordinator + participant Redis as RedisSearchStateStore + participant Dispatcher as SupplierTaskDispatcher + participant Supplier as 多个供应商 + participant CallbackAPI as SupplierCallbackRpcService + participant Callback as SupplierCallbackService + participant Waiter as LocalSearchWaiters + + Caller->>SearchAPI: asyncSearch(request) + SearchAPI->>Coordinator: 执行查询 + Coordinator->>Redis: 初始化状态、待完成集合和 TTL + Coordinator->>Waiter: 注册本地等待器 + Coordinator->>Dispatcher: 并发分发供应商任务 + Dispatcher-->>Supplier: 异步查询 + + loop 供应商分批返回结果 + Supplier->>CallbackAPI: callback(result, searchFinished) + CallbackAPI->>Callback: 处理回调 + Callback->>Redis: 追加结果并更新待完成集合 + end + + Redis-->>Waiter: 全部完成后发布通知 + Waiter-->>Coordinator: 提前唤醒 + Coordinator->>Redis: 读取最终状态和结果 + Coordinator-->>SearchAPI: PolicySearchResponse + SearchAPI-->>Caller: 完成异步调用 +``` + +本地等待器和 Redis Pub/Sub 只用于提前唤醒。即使通知丢失,查询线程仍会定期检查 Redis。超过总超时时间后, +Lua 脚本会原子地将查询状态更新为 `TIMED_OUT`,返回已经收到的部分结果,并拒绝迟到回调继续修改结果。 + +## 场景二:海量政策匹配与快照切换 + +每个政策快照使用独立目录保存 RocksDB 明细库,并在内存中维护 RoaringBitmap 匹配索引。新版本先在旁路 +完成全量加载、增量追平和一致性校验,只有通过校验后才会替换当前服务版本。 + +```mermaid +flowchart TD + Start["应用启动或运行时刷新"] --> Create["创建版本隔离目录"] + Create --> Full["加载全量政策"] + Full --> Position["记录全量数据切点"] + Position --> Replay["回放切点后的增量消息"] + Replay --> CaughtUp{"已追平最新位置?"} + CaughtUp -- 否 --> Replay + CaughtUp -- 是 --> Validate["校验明细、索引和消息位置"] + Validate --> Valid{"校验成功?"} + Valid -- 否 --> Discard["关闭并删除候选快照"] + Valid -- 是 --> Activate["原子激活新快照"] + Activate --> NewQuery["新查询使用新版本"] + Activate --> Drain["旧版本等待已有查询释放租约"] + Drain --> Delete["关闭并删除旧快照"] +``` + +查询通过 `ActiveSnapshotRegistry.acquire()` 获取快照租约,并使用 `try-with-resources` 释放。切换完成后, +新查询立即使用新快照,旧快照则保留到最后一个旧查询结束,从而避免正在执行的查询访问已关闭的 RocksDB。 + +快照方案的主要扩展点: + +1. `FullPolicyLoader`:加载全量政策,并返回全量数据对应的消息位置。 +2. `IncrementalReplayer`:回放全量切点之后的新增、修改和删除事件。 +3. `SnapshotValidator`:校验政策明细、位图索引和消息位置的一致性。 +4. `SnapshotDirectory`:隔离不同快照版本的存储目录。 +5. `PolicySnapshotService`:负责首次初始化、失败重试和运行时刷新。 + +Kafka 政策变更消息使用全局单调递增的 `position`,重复或乱序消息会被忽略。如果 Topic 使用多个分区, +生产端必须提供全局序列;否则应将当前位置模型调整为按分区保存 offset。 + +## 场景三:幂等下单与订单状态机 + +`BookOrderRpcService.createOrder` 以调用方生成的 `requestId` 作为幂等键。下单首先保存 `CREATE` 状态订单, +提交后才由 Seata Saga 调用 GDS,因此不会在本地数据库事务中持有远程调用。 + +```mermaid +sequenceDiagram + autonumber + participant Caller as 调用方 + participant Order as 下单应用服务 + participant DB as 本地数据库 + participant Saga as Seata Saga + participant GDS as GDS + participant Task as 任务/补偿表 + + Caller->>Order: createOrder(requestId) + Order->>Order: 查询 Redis 幂等结果缓存 + Order->>Order: Redisson RLock 获取 requestId 短锁 + Order->>Order: 锁内二次查缓存,仅持锁者查数据库 + Order->>DB: 本地事务写订单 CREATE + DB-->>Order: 提交成功 + Order->>Order: 回填结果缓存并释放 RLock + Order->>DB: 抢占订单创建调度租约 + Order->>Saga: 启动 BookOrderCreationSaga + Saga->>GDS: 发起 PNR 占编 + GDS-->>Saga: SUCCESS / FAIL / UNKNOWN + alt SUCCESS + Saga->>DB: CREATE_SUCCEEDED → WAIT_PAY + Saga->>Task: 幂等创建待支付任务 + else FAIL + Saga->>DB: CREATE_FAILED → CREATE_FAIL + Saga->>Task: 幂等创建促销库存返还任务 + else UNKNOWN + Saga->>DB: START_VALIDATE → VALIDATING + Saga->>Task: 创建 GDS 核对任务 + 人工任务 + end +``` + +订单与 `START_ORDER_CREATION` Outbox 在同一本地事务中提交。事务后立即尝试发布;如果进程在订单提交后、 +启动 Saga 前崩溃,`OrderCreationOutboxScheduler` 会重新投递未发布消息。Saga 使用 `orderNo` 作为业务幂等键, +已经启动的流程由 Seata 根据持久化的状态机日志继续恢复。 +Saga 使用 `orderNo` 作为业务幂等键;GDS 侧也必须使用订单号或稳定请求号保证占编幂等。 + +高并发重复请求首先读取 `book:create:result:{requestId}` 幂等结果缓存,命中后不访问数据库。缓存未命中时, +使用 Redisson `RLock` 锁定 `book:create:lock:{requestId}`,锁内再次检查缓存,只有锁持有者才允许查询数据库和 +执行本地创建事务;未获得锁的请求等待首个请求回填结果缓存,同样不会查询数据库。RLock 最多等待 300ms, +持锁期间由 Redisson watchdog 自动续期;锁只覆盖本地事务,不覆盖耗时不可控的 Saga/GDS 调用。 + +订单状态迁移后会同步刷新结果缓存,缓存 TTL 为 10 分钟。Redis 不可用时锁和缓存主动降级,最终仍由数据库 +`request_id` 唯一索引保证只能创建一张订单;Redisson 锁和结果缓存是防穿透、削峰层,不是最终一致性依据。 + +外部系统通过 `BookOrderRpcService.fireEvent` 提交支付、出票、验真、取消等业务事件,不能直接指定目标状态。 +`OrderStateMachine` 使用 `(当前状态, 业务事件)` 定位唯一迁移,状态参考 `ipolicytradecore`: + +```mermaid +stateDiagram-v2 + CREATE --> WAIT_PAY: CREATE_SUCCEEDED + CREATE --> CREATE_FAIL: CREATE_FAILED + CREATE --> VALIDATING: START_VALIDATE + CREATE --> CANCEL: CANCEL + WAIT_PAY --> BOOKING: PAY_SUCCEEDED + WAIT_PAY --> CANCEL: CANCEL + BOOKING --> BOOKED: BOOK_SUCCEEDED + BOOKING --> BOOK_FAIL: BOOK_FAILED + BOOKING --> VALIDATING: START_VALIDATE + BOOKED --> VALIDATING: START_VALIDATE + BOOKED --> CANCEL: CANCEL + BOOKED --> REFUNDED: REFUND_SUCCEEDED + BOOK_FAIL --> VALIDATING: START_VALIDATE + BOOK_FAIL --> CANCEL: CANCEL + VALIDATING --> BOOKED: VALIDATE_SUCCEEDED + VALIDATING --> VALIDATE_FAIL: VALIDATE_FAILED + VALIDATING --> WAIT_PAY: GDS_BOOKING_CONFIRMED + VALIDATING --> CREATE_FAIL: GDS_BOOKING_REJECTED + VALIDATE_FAIL --> BOOKED: VALIDATE_SUCCEEDED + VALIDATE_FAIL --> CANCEL: CANCEL + CREATE_FAIL --> DELETED: DELETE + BOOK_FAIL --> DELETED: DELETE + CANCEL --> DELETED: DELETE + REFUNDED --> DELETED: DELETE +``` + +一次迁移依次执行 Guard、前置 Action、领域状态变更、原子持久化和后置 Action。`PAY_SUCCEEDED`、 +`BOOK_SUCCEEDED` 等关键事件通过 Guard 校验支付单号、PNR 等业务凭据;业务可以通过迁移 Builder 注册更多 +风控校验、库存检查和任务创建处理器,而不修改状态机引擎。 + +状态事件携带全局唯一 `eventId` 和 `expectedVersion`。仓储在一个事务边界内完成以下写入: + +1. 使用 `order_no + version` compare-and-set 更新订单,阻止并发覆盖。 +2. 保存 `eventId` 处理结果,重复消息直接返回第一次处理的快照。 +3. 追加包含 `from/event/to/operator` 的完整状态历史。 +4. 写入 Outbox 消息,由独立发布任务可靠投递给库存、支付、出票等下游。 + +后置 Action 仅在事务提交后执行,失败会进入 `FailedPostActionStore` 等待重试,不会把已提交订单回滚成旧状态。 +示例使用内存实现展示原子语义;生产落地应使用数据库唯一索引、条件更新、状态历史表、Outbox 表和重试任务, +并由消息消费方继续按照 `eventId` 幂等。 + +### Seata Saga 与补偿 + +`statelang/book_order_creation_saga.json` 是 Seata 状态语言定义,包含 GDS 服务节点、结果 Choice、异常捕获和 +`CancelReservedPnr` 补偿节点。`SeataOrderCreationSaga` 使用 `orderNo` 作为 business key 启动状态机; +`SeataSagaConfiguration` 在存在 `DataSource` 时创建数据库持久化的 `DbStateMachineConfig` 和 +`StateMachineEngine`。没有引擎 Bean 的本地演示环境使用相同 Saga State Services 直接编排,业务行为一致。 + +Saga 提供的是可恢复的最终一致性,而不是把 GDS 变成支持 ACID 回滚的数据库资源。每个外部正向动作都必须有 +幂等补偿语义,并允许空补偿: + +- PNR 占编失败且已经扣减促销库存:创建 `RETURN_PROMOTION_STOCK` 任务。 +- PNR 已占编但采购商取消:创建 `CANCEL_PNR` 任务。 +- 支付成功但最终出票失败:创建 `REFUND_PAYMENT` 任务。 +- GDS 返回 UNKNOWN:创建 `VALIDATE_GDS_BOOKING` 与人工核对任务,定时比较本地状态和 GDS 实际状态。 + +补偿表使用业务唯一键防重;取消 PNR 额外使用 `(order_no, child_order_no, pnr, task_type)` 唯一索引。任务采用 +指数退避重试,超过五次进入 `MANUAL_REQUIRED`。示例 DDL 位于 `db/book_order_schema.sql`;Seata Server 和 +Saga 引擎日志表应使用部署版本对应的官方脚本创建,避免跨版本复制表结构。 + +## 新增架构场景的约定 + +新增交易、下单或其他架构伪代码时,应遵循以下约定: + +1. 在 README 的场景目录中说明要解决的问题、关键约束和方案状态。 +2. 对外契约放在 `api.`,内部实现放在对应的 `` 业务包。 +3. 使用 `application/domain/infrastructure` 表达职责边界,不让领域规则依赖具体中间件。 +4. 优先提供展示关键协作关系的最小实现,避免把伪代码扩展成不完整的生产框架。 +5. 用精简的流程代码和注释表达幂等、一致性、并发、超时、切换和失败隔离等关键行为。 +6. 在场景文档中记录设计取舍、适用边界,以及生产落地仍需补充的能力。 diff --git a/architecture/pom.xml b/architecture/pom.xml new file mode 100644 index 0000000..e677418 --- /dev/null +++ b/architecture/pom.xml @@ -0,0 +1,84 @@ + + + 4.0.0 + + com.arch + architecture + 1.0-SNAPSHOT + + + 8 + 8 + UTF-8 + 2.7.18 + 3.2.15 + 1.8.0 + + 3.24.3 + + + + + org.rocksdb + rocksdbjni + 8.11.3 + + + org.roaringbitmap + RoaringBitmap + 0.9.47 + + + org.springframework.boot + spring-boot-starter + ${spring-boot.version} + + + org.redisson + redisson + ${redisson.version} + + + org.springframework.boot + spring-boot-starter-data-redis + ${spring-boot.version} + + + org.springframework.kafka + spring-kafka + 2.9.13 + + + com.fasterxml.jackson.core + jackson-databind + 2.13.5 + + + org.apache.dubbo + dubbo-spring-boot-starter + ${dubbo.version} + + + io.seata + seata-saga-engine + ${seata.version} + + + io.seata + seata-saga-engine-store + ${seata.version} + + + + + + + org.springframework.boot + spring-boot-maven-plugin + ${spring-boot.version} + + + + diff --git a/architecture/src/main/java/com/arch/policy/PolicySearchApplication.java b/architecture/src/main/java/com/arch/policy/PolicySearchApplication.java new file mode 100644 index 0000000..9bdfd8b --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/PolicySearchApplication.java @@ -0,0 +1,15 @@ +package com.arch.policy; + +import org.apache.dubbo.config.spring.context.annotation.EnableDubbo; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.kafka.annotation.EnableKafka; + +@EnableKafka +@EnableDubbo +@SpringBootApplication +public class PolicySearchApplication { + public static void main(String[] args) { + SpringApplication.run(PolicySearchApplication.class, args); + } +} diff --git a/architecture/src/main/java/com/arch/policy/api/order/OrderRpcService.java b/architecture/src/main/java/com/arch/policy/api/order/OrderRpcService.java new file mode 100644 index 0000000..2b2c840 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/api/order/OrderRpcService.java @@ -0,0 +1,14 @@ +package com.arch.policy.api.order; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.OrderResponse; + +/** + * @Author : haiyang.luo + * @Date : 2026/7/22 15:44 + * @Description : + */ +public interface OrderRpcService { + + OrderResponse createOrder(OrderRequest orderRequest); +} diff --git a/architecture/src/main/java/com/arch/policy/api/search/PolicySearchRpcService.java b/architecture/src/main/java/com/arch/policy/api/search/PolicySearchRpcService.java new file mode 100644 index 0000000..98533cf --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/api/search/PolicySearchRpcService.java @@ -0,0 +1,10 @@ +package com.arch.policy.api.search; + +import com.arch.policy.common.search.PolicySearchRequest; +import com.arch.policy.common.search.PolicySearchResponse; + +import java.util.concurrent.CompletableFuture; + +public interface PolicySearchRpcService { + CompletableFuture asyncSearch(PolicySearchRequest request); +} diff --git a/architecture/src/main/java/com/arch/policy/api/search/SupplierCallbackRpcService.java b/architecture/src/main/java/com/arch/policy/api/search/SupplierCallbackRpcService.java new file mode 100644 index 0000000..8995d25 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/api/search/SupplierCallbackRpcService.java @@ -0,0 +1,8 @@ +package com.arch.policy.api.search; + +import com.arch.policy.common.search.CallbackResponse; +import com.arch.policy.common.search.SupplierCallbackRequest; + +public interface SupplierCallbackRpcService { + CallbackResponse callback(SupplierCallbackRequest request); +} diff --git a/architecture/src/main/java/com/arch/policy/common/config/PolicySearchConfiguration.java b/architecture/src/main/java/com/arch/policy/common/config/PolicySearchConfiguration.java new file mode 100644 index 0000000..21191fb --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/config/PolicySearchConfiguration.java @@ -0,0 +1,109 @@ +package com.arch.policy.common.config; + +import com.arch.policy.search.application.AsyncSearchCoordinator; +import com.arch.policy.search.application.LocalSearchWaiters; +import com.arch.policy.search.application.PolicyIncrementalUpdater; +import com.arch.policy.search.application.PolicyStartupRunner; +import com.arch.policy.search.application.SearchStateStore; +import com.arch.policy.search.application.SupplierCallbackService; +import com.arch.policy.search.application.SupplierTaskDispatcher; +import com.arch.policy.search.domain.snapshot.ActiveSnapshotRegistry; +import com.arch.policy.search.domain.snapshot.DefaultSnapshotValidator; +import com.arch.policy.search.domain.snapshot.FileSystemSnapshotDirectory; +import com.arch.policy.search.domain.snapshot.PolicySnapshotService; +import com.arch.policy.search.domain.snapshot.SnapshotBuilder; +import com.arch.policy.search.domain.snapshot.SnapshotPorts.FullPolicyLoader; +import com.arch.policy.search.domain.snapshot.SnapshotPorts.IncrementalReplayer; +import com.arch.policy.search.infrastructure.demo.DemoSupplierTaskDispatcher; +import com.arch.policy.search.infrastructure.kafka.KafkaPolicyChangeListener; +import com.arch.policy.search.infrastructure.redis.RedisSearchStateStore; +import com.arch.policy.search.infrastructure.redis.SearchFinishedSubscriber; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.data.redis.connection.RedisConnectionFactory; +import org.springframework.data.redis.listener.ChannelTopic; +import org.springframework.data.redis.listener.RedisMessageListenerContainer; +import org.springframework.data.redis.core.StringRedisTemplate; + +import java.nio.file.Paths; +import java.util.concurrent.Executor; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; + +@Configuration +public class PolicySearchConfiguration { + @Bean public ActiveSnapshotRegistry activeSnapshotRegistry() { return new ActiveSnapshotRegistry(); } + + @Bean public PolicyIncrementalUpdater policyIncrementalUpdater(ActiveSnapshotRegistry registry) { + return new PolicyIncrementalUpdater(registry); + } + + @Bean(destroyMethod = "shutdown") public Executor policySearchExecutor() { + return Executors.newFixedThreadPool(Math.max(2, Runtime.getRuntime().availableProcessors())); + } + + @Bean public ObjectMapper objectMapper() { return new ObjectMapper(); } + + @Bean public SearchStateStore searchStateStore(StringRedisTemplate redis) { + return new RedisSearchStateStore(redis); + } + + @Bean public LocalSearchWaiters localSearchWaiters() { return new LocalSearchWaiters(); } + + @Bean public SupplierCallbackService supplierCallbackService( + SearchStateStore store, ObjectMapper mapper, + @Value("${policy.search.redis-ttl-seconds:60}") long ttlSeconds) { + return new SupplierCallbackService(store, mapper, ttlSeconds); + } + + @Bean(destroyMethod = "shutdown") public ScheduledExecutorService demoSupplierExecutor() { + return Executors.newScheduledThreadPool(4); + } + + @Bean public SupplierTaskDispatcher supplierTaskDispatcher( + ScheduledExecutorService demoSupplierExecutor, SupplierCallbackService callbackService) { + return new DemoSupplierTaskDispatcher(demoSupplierExecutor, callbackService); + } + + @Bean public AsyncSearchCoordinator asyncSearchCoordinator( + SearchStateStore store, SupplierTaskDispatcher dispatcher, LocalSearchWaiters waiters, + ObjectMapper mapper, @Value("${policy.search.redis-ttl-seconds:60}") long ttlSeconds) { + return new AsyncSearchCoordinator(store, dispatcher, waiters, mapper, ttlSeconds); + } + + @Bean public RedisMessageListenerContainer searchFinishedListener( + RedisConnectionFactory connectionFactory, LocalSearchWaiters waiters) { + RedisMessageListenerContainer container = new RedisMessageListenerContainer(); + container.setConnectionFactory(connectionFactory); + container.addMessageListener(new SearchFinishedSubscriber(waiters), + new ChannelTopic(RedisSearchStateStore.FINISHED_CHANNEL)); + return container; + } + + @Bean public KafkaPolicyChangeListener kafkaPolicyChangeListener(ObjectMapper mapper, + PolicyIncrementalUpdater updater) { + return new KafkaPolicyChangeListener(mapper, updater); + } + + @Bean public SnapshotBuilder snapshotBuilder(FullPolicyLoader fullLoader, + IncrementalReplayer replayer, + @Value("${policy.snapshot.directory:./data/policy-snapshots}") String directory) { + return new SnapshotBuilder(fullLoader, replayer, new DefaultSnapshotValidator(), + new FileSystemSnapshotDirectory(Paths.get(directory))); + } + + @Bean(destroyMethod = "close") public PolicySnapshotService policySnapshotService( + SnapshotBuilder builder, ActiveSnapshotRegistry registry, + @Value("${policy.snapshot.retry-delay-ms:5000}") long retryDelayMillis) { + return new PolicySnapshotService(builder, registry, + Executors.newSingleThreadScheduledExecutor(), retryDelayMillis); + } + + @Bean public PolicyStartupRunner policyStartupRunner( + PolicySnapshotService service, + @Value("${policy.snapshot.initial-version:startup}") String initialVersion) { + return new PolicyStartupRunner(service, initialVersion); + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/config/RedissonConfiguration.java b/architecture/src/main/java/com/arch/policy/common/config/RedissonConfiguration.java new file mode 100644 index 0000000..5fb5cc6 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/config/RedissonConfiguration.java @@ -0,0 +1,28 @@ +package com.arch.policy.common.config; + +import org.redisson.Redisson; +import org.redisson.api.RedissonClient; +import org.redisson.config.Config; +import org.redisson.config.SingleServerConfig; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class RedissonConfiguration { + @Bean(destroyMethod = "shutdown") + @ConditionalOnMissingBean(RedissonClient.class) + public RedissonClient redissonClient( + @Value("${spring.redis.host:localhost}") String host, + @Value("${spring.redis.port:6379}") int port, + @Value("${spring.redis.database:0}") int database, + @Value("${spring.redis.password:}") String password) { + Config config = new Config(); + SingleServerConfig server = config.useSingleServer() + .setAddress("redis://" + host + ":" + port) + .setDatabase(database); + if (password != null && !password.trim().isEmpty()) server.setPassword(password); + return Redisson.create(config); + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/config/SeataSagaConfiguration.java b/architecture/src/main/java/com/arch/policy/common/config/SeataSagaConfiguration.java new file mode 100644 index 0000000..94cd985 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/config/SeataSagaConfiguration.java @@ -0,0 +1,54 @@ +package com.arch.policy.common.config; + +import io.seata.saga.engine.StateMachineEngine; +import io.seata.saga.engine.config.DbStateMachineConfig; +import io.seata.saga.engine.impl.ProcessCtrlStateMachineEngine; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +import javax.sql.DataSource; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; + +@Configuration +public class SeataSagaConfiguration { + @Bean(destroyMethod = "shutdown") + @ConditionalOnBean(DataSource.class) + public ThreadPoolExecutor seataSagaExecutor() { + return new ThreadPoolExecutor(2, 16, 60L, TimeUnit.SECONDS, + new LinkedBlockingQueue(1000), new ThreadPoolExecutor.CallerRunsPolicy()); + } + + @Bean + @ConditionalOnBean(DataSource.class) + public DbStateMachineConfig dbStateMachineConfig( + DataSource dataSource, ThreadPoolExecutor seataSagaExecutor, + @Value("${spring.application.name:architecture}") String applicationId, + @Value("${order.seata.tx-service-group:order-saga-group}") String txServiceGroup, + @Value("${order.seata.tenant-id:order}") String tenantId) { + DbStateMachineConfig config = new DbStateMachineConfig(); + config.setDataSource(dataSource); + config.setApplicationId(applicationId); + config.setTxServiceGroup(txServiceGroup); + config.setDefaultTenantId(tenantId); + config.setThreadPoolExecutor(seataSagaExecutor); + config.setAutoRegisterResources(true); + config.setResources(new String[] { "classpath*:statelang/order_creation_saga.json" }); + config.setSagaJsonParser("jackson"); + config.setSagaBranchRegisterEnable(true); + return config; + } + + @Bean + @ConditionalOnBean(DbStateMachineConfig.class) + @ConditionalOnMissingBean(StateMachineEngine.class) + public StateMachineEngine stateMachineEngine(DbStateMachineConfig config) { + ProcessCtrlStateMachineEngine engine = new ProcessCtrlStateMachineEngine(); + engine.setStateMachineConfig(config); + return engine; + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonDistributedLock.java b/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonDistributedLock.java new file mode 100644 index 0000000..810190c --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonDistributedLock.java @@ -0,0 +1,47 @@ +package com.arch.policy.common.infrastructure.redis; + +import com.arch.policy.common.lock.DistributedLock; +import com.arch.policy.common.lock.DistributedLockUnavailableException; +import org.redisson.api.RLock; +import org.redisson.api.RedissonClient; +import org.redisson.client.RedisException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.TimeUnit; + +public final class RedissonDistributedLock implements DistributedLock { + private static final Logger LOGGER = LoggerFactory.getLogger(RedissonDistributedLock.class); + private final RedissonClient redisson; + + public RedissonDistributedLock(RedissonClient redisson) { this.redisson = redisson; } + + @Override public LockHandle tryAcquire(String key, long waitMillis) { + RLock lock = redisson.getLock(key); + try { + // 不指定 leaseTime,交由 Redisson watchdog 在持锁线程存活期间自动续期。 + boolean acquired = lock.tryLock(waitMillis, TimeUnit.MILLISECONDS); + return acquired ? new RedissonLockHandle(lock) : null; + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + return null; + } catch (RedisException redisFailure) { + throw new DistributedLockUnavailableException( + "Redisson lock unavailable, key=" + key, redisFailure); + } + } + + private static final class RedissonLockHandle implements LockHandle { + private final RLock lock; + + private RedissonLockHandle(RLock lock) { this.lock = lock; } + + @Override public void close() { + try { + if (lock.isHeldByCurrentThread()) lock.unlock(); + } catch (RedisException redisFailure) { + LOGGER.warn("Failed to release Redisson lock, key={}", lock.getName(), redisFailure); + } + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonRedisClient.java b/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonRedisClient.java new file mode 100644 index 0000000..599ce29 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/infrastructure/redis/RedissonRedisClient.java @@ -0,0 +1,30 @@ +package com.arch.policy.common.infrastructure.redis; + +import com.arch.policy.common.redis.RedisClient; +import com.arch.policy.common.redis.RedisClientException; +import org.redisson.api.RedissonClient; +import org.redisson.client.RedisException; + +import java.util.concurrent.TimeUnit; + +public final class RedissonRedisClient implements RedisClient { + private final RedissonClient redisson; + + public RedissonRedisClient(RedissonClient redisson) { this.redisson = redisson; } + + @Override public T get(String key) { + try { + return redisson.getBucket(key).get(); + } catch (RedisException redisFailure) { + throw new RedisClientException("Failed to read Redis key=" + key, redisFailure); + } + } + + @Override public void set(String key, Object value, long ttlMillis) { + try { + redisson.getBucket(key).set(value, ttlMillis, TimeUnit.MILLISECONDS); + } catch (RedisException redisFailure) { + throw new RedisClientException("Failed to write Redis key=" + key, redisFailure); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/lock/DistributedLock.java b/architecture/src/main/java/com/arch/policy/common/lock/DistributedLock.java new file mode 100644 index 0000000..dbd5277 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/lock/DistributedLock.java @@ -0,0 +1,14 @@ +package com.arch.policy.common.lock; + +public interface DistributedLock { + /** + * 尝试获取指定 key 对应的分布式锁。 + * + * @return 锁句柄;等待超时返回 null + */ + LockHandle tryAcquire(String key, long waitMillis); + + interface LockHandle extends AutoCloseable { + @Override void close(); + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/lock/DistributedLockUnavailableException.java b/architecture/src/main/java/com/arch/policy/common/lock/DistributedLockUnavailableException.java new file mode 100644 index 0000000..1e8828f --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/lock/DistributedLockUnavailableException.java @@ -0,0 +1,7 @@ +package com.arch.policy.common.lock; + +public final class DistributedLockUnavailableException extends RuntimeException { + public DistributedLockUnavailableException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/model/MessagePosition.java b/architecture/src/main/java/com/arch/policy/common/model/MessagePosition.java new file mode 100644 index 0000000..edca20b --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/model/MessagePosition.java @@ -0,0 +1,20 @@ +package com.arch.policy.common.model; + +public final class MessagePosition implements Comparable { + public static final MessagePosition BEGINNING = new MessagePosition(0); + private final long value; + + public MessagePosition(long value) { + if (value < 0) throw new IllegalArgumentException("position must be non-negative"); + this.value = value; + } + + public long getValue() { return value; } + + @Override public int compareTo(MessagePosition other) { return Long.compare(value, other.value); } + @Override public boolean equals(Object other) { + return other instanceof MessagePosition && value == ((MessagePosition) other).value; + } + @Override public int hashCode() { return Long.valueOf(value).hashCode(); } + @Override public String toString() { return Long.toString(value); } +} diff --git a/architecture/src/main/java/com/arch/policy/common/model/PolicyChange.java b/architecture/src/main/java/com/arch/policy/common/model/PolicyChange.java new file mode 100644 index 0000000..b014367 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/model/PolicyChange.java @@ -0,0 +1,30 @@ +package com.arch.policy.common.model; + +public final class PolicyChange { + public enum Type { UPSERT, DELETE } + + private final Type type; + private final int policyId; + private final PolicyRecord policy; + private final MessagePosition position; + + private PolicyChange(Type type, int policyId, PolicyRecord policy, MessagePosition position) { + this.type = type; + this.policyId = policyId; + this.policy = policy; + this.position = position; + } + + public static PolicyChange upsert(PolicyRecord policy, MessagePosition position) { + return new PolicyChange(Type.UPSERT, policy.getId(), policy, position); + } + + public static PolicyChange delete(int policyId, MessagePosition position) { + return new PolicyChange(Type.DELETE, policyId, null, position); + } + + public Type getType() { return type; } + public int getPolicyId() { return policyId; } + public PolicyRecord getPolicy() { return policy; } + public MessagePosition getPosition() { return position; } +} diff --git a/architecture/src/main/java/com/arch/policy/common/model/PolicyRecord.java b/architecture/src/main/java/com/arch/policy/common/model/PolicyRecord.java new file mode 100644 index 0000000..6fbab70 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/model/PolicyRecord.java @@ -0,0 +1,27 @@ +package com.arch.policy.common.model; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.Set; + +public final class PolicyRecord { + private final int id; + private final byte[] detail; + private final Set indexTerms; + + public PolicyRecord(int id, byte[] detail, Set indexTerms) { + if (id < 0) { + throw new IllegalArgumentException("policy id must be non-negative"); + } + this.id = id; + this.detail = Arrays.copyOf(detail, detail.length); + this.indexTerms = Collections.unmodifiableSet(new HashSet(indexTerms)); + } + + public int getId() { return id; } + + public byte[] getDetail() { return Arrays.copyOf(detail, detail.length); } + + public Set getIndexTerms() { return indexTerms; } +} diff --git a/architecture/src/main/java/com/arch/policy/common/redis/RedisClient.java b/architecture/src/main/java/com/arch/policy/common/redis/RedisClient.java new file mode 100644 index 0000000..a8a2f88 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/redis/RedisClient.java @@ -0,0 +1,7 @@ +package com.arch.policy.common.redis; + +public interface RedisClient { + T get(String key); + + void set(String key, Object value, long ttlMillis); +} diff --git a/architecture/src/main/java/com/arch/policy/common/redis/RedisClientException.java b/architecture/src/main/java/com/arch/policy/common/redis/RedisClientException.java new file mode 100644 index 0000000..15dfdcf --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/redis/RedisClientException.java @@ -0,0 +1,7 @@ +package com.arch.policy.common.redis; + +public final class RedisClientException extends RuntimeException { + public RedisClientException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/architecture/src/main/java/com/arch/policy/common/search/CallbackResponse.java b/architecture/src/main/java/com/arch/policy/common/search/CallbackResponse.java new file mode 100644 index 0000000..f5fccf7 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/search/CallbackResponse.java @@ -0,0 +1,10 @@ +package com.arch.policy.common.search; + +import java.io.Serializable; + +public final class CallbackResponse implements Serializable { + private static final long serialVersionUID = 1L; + private final boolean success; + public CallbackResponse(boolean success) { this.success = success; } + public boolean isSuccess() { return success; } +} diff --git a/architecture/src/main/java/com/arch/policy/common/search/PolicySearchRequest.java b/architecture/src/main/java/com/arch/policy/common/search/PolicySearchRequest.java new file mode 100644 index 0000000..6553673 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/search/PolicySearchRequest.java @@ -0,0 +1,22 @@ +package com.arch.policy.common.search; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +public final class PolicySearchRequest implements Serializable { + private static final long serialVersionUID = 1L; + private String criteria; + private List supplierIds = new ArrayList(); + private long totalTimeoutMillis = 3000; + + public String getCriteria() { return criteria; } + public void setCriteria(String criteria) { this.criteria = criteria; } + public List getSupplierIds() { return Collections.unmodifiableList(supplierIds); } + public void setSupplierIds(List supplierIds) { + this.supplierIds = supplierIds == null ? new ArrayList() : new ArrayList(supplierIds); + } + public long getTotalTimeoutMillis() { return totalTimeoutMillis; } + public void setTotalTimeoutMillis(long totalTimeoutMillis) { this.totalTimeoutMillis = totalTimeoutMillis; } +} diff --git a/architecture/src/main/java/com/arch/policy/common/search/PolicySearchResponse.java b/architecture/src/main/java/com/arch/policy/common/search/PolicySearchResponse.java new file mode 100644 index 0000000..fa0d708 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/search/PolicySearchResponse.java @@ -0,0 +1,22 @@ +package com.arch.policy.common.search; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +public final class PolicySearchResponse implements Serializable { + private static final long serialVersionUID = 1L; + private final String searchKey; + private final String state; + private final List results; + + public PolicySearchResponse(String searchKey, String state, List results) { + this.searchKey = searchKey; + this.state = state; + this.results = Collections.unmodifiableList(new ArrayList(results)); + } + public String getSearchKey() { return searchKey; } + public String getState() { return state; } + public List getResults() { return results; } +} diff --git a/architecture/src/main/java/com/arch/policy/common/search/SupplierCallbackRequest.java b/architecture/src/main/java/com/arch/policy/common/search/SupplierCallbackRequest.java new file mode 100644 index 0000000..a4b5ed2 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/common/search/SupplierCallbackRequest.java @@ -0,0 +1,25 @@ +package com.arch.policy.common.search; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +public final class SupplierCallbackRequest implements Serializable { + private static final long serialVersionUID = 1L; + private String searchKey; + private String supplierId; + private List results = new ArrayList(); + private boolean searchFinished; + + public String getSearchKey() { return searchKey; } + public void setSearchKey(String searchKey) { this.searchKey = searchKey; } + public String getSupplierId() { return supplierId; } + public void setSupplierId(String supplierId) { this.supplierId = supplierId; } + public List getResults() { return Collections.unmodifiableList(results); } + public void setResults(List results) { + this.results = results == null ? new ArrayList() : new ArrayList(results); + } + public boolean isSearchFinished() { return searchFinished; } + public void setSearchFinished(boolean searchFinished) { this.searchFinished = searchFinished; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/cache/OrderCacheUtil.java b/architecture/src/main/java/com/arch/policy/order/cache/OrderCacheUtil.java new file mode 100644 index 0000000..6a2e363 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/cache/OrderCacheUtil.java @@ -0,0 +1,47 @@ +package com.arch.policy.order.cache; + +import com.arch.policy.order.model.OrderResponse; +import org.redisson.api.RedissonClient; +import org.redisson.client.RedisException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +/** + * @Author : haiyang.luo + * @Date : 2026/7/22 15:58 + * @Description : + */ +@Component +public class OrderCacheUtil { + + private static final Logger LOGGER = LoggerFactory.getLogger(OrderCacheUtil.class); + + @Autowired + private RedissonClient redissonClient; + + public OrderResponse getCache(String orderSerialNo) { + try { + return redissonClient.getBucket(cacheKey(orderSerialNo)).get(); + } catch (RedisException redisFailure) { + LOGGER.warn("Create-order result cache unavailable, orderSerialNo={}", orderSerialNo, + redisFailure); + return null; + } + } + + public void setCache(String orderSerialNo, OrderResponse response) { + try { + redissonClient.getBucket(cacheKey(orderSerialNo)).set(response, 600_000L, + java.util.concurrent.TimeUnit.MILLISECONDS); + } catch (RedisException redisFailure) { + LOGGER.warn("Create-order result cache unavailable, orderSerialNo={}", orderSerialNo, + redisFailure); + } + } + + private String cacheKey(String orderSerialNo) { + return "order:create:result:" + orderSerialNo; + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/cache/OrderLockUtil.java b/architecture/src/main/java/com/arch/policy/order/cache/OrderLockUtil.java new file mode 100644 index 0000000..2f1bd36 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/cache/OrderLockUtil.java @@ -0,0 +1,45 @@ +package com.arch.policy.order.cache; + +import org.redisson.api.RLock; +import org.redisson.api.RedissonClient; +import org.redisson.client.RedisException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import javax.annotation.Resource; +import java.util.concurrent.TimeUnit; + +@Component +public class OrderLockUtil { + private static final Logger LOGGER = LoggerFactory.getLogger(OrderLockUtil.class); + + @Resource + private RedissonClient redissonClient; + + public boolean tryLock(String orderSerialNo) { + RLock lock = redissonClient.getLock("order:create:lock:" + orderSerialNo); + try { + // 不设置固定过期时间,使用Redisson watchdog自动续期;只锁本地建单,不锁三方调用。 + return lock.tryLock(300L, TimeUnit.MILLISECONDS); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + return false; + } catch (RedisException redisFailure) { + LOGGER.warn("Create-order lock unavailable, orderSerialNo={}", orderSerialNo, + redisFailure); + // Redis故障时继续执行,由订单库唯一索引保证最终幂等。 + return true; + } + } + + public void unlock(String orderSerialNo) { + RLock lock = redissonClient.getLock("order:create:lock:" + orderSerialNo); + try { + if (lock.isHeldByCurrentThread()) lock.unlock(); + } catch (RedisException redisFailure) { + LOGGER.warn("Release create-order lock failed, orderSerialNo={}", orderSerialNo, + redisFailure); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/InventoryTransaction.java b/architecture/src/main/java/com/arch/policy/order/model/InventoryTransaction.java new file mode 100644 index 0000000..40231bc --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/InventoryTransaction.java @@ -0,0 +1,23 @@ +package com.arch.policy.order.model; + +/** 库存库中的幂等扣减流水。 */ +public class InventoryTransaction { + private final String tradeOrderSerialNo; + private final String skuId; + private final int quantity; + private InventoryTransactionStatus status; + + public InventoryTransaction(String tradeOrderSerialNo, String skuId, int quantity, + InventoryTransactionStatus status) { + this.tradeOrderSerialNo = tradeOrderSerialNo; + this.skuId = skuId; + this.quantity = quantity; + this.status = status; + } + + public String getTradeOrderSerialNo() { return tradeOrderSerialNo; } + public String getSkuId() { return skuId; } + public int getQuantity() { return quantity; } + public InventoryTransactionStatus getStatus() { return status; } + public void setStatus(InventoryTransactionStatus status) { this.status = status; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/InventoryTransactionStatus.java b/architecture/src/main/java/com/arch/policy/order/model/InventoryTransactionStatus.java new file mode 100644 index 0000000..e55c393 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/InventoryTransactionStatus.java @@ -0,0 +1,25 @@ +package com.arch.policy.order.model; + +/** 库存库流水状态;库存库是该状态的权威来源。 */ +public enum InventoryTransactionStatus { + /** 正在执行库存扣减,最终结果尚未确定。 */ + DEDUCTING, + + /** 库存已经扣减,等待订单创建结果决定确认或返还。 */ + DEDUCTED, + + /** 三方订单创建成功,库存扣减已经最终确认。 */ + CONFIRMED, + + /** 正在执行库存返还,最终结果尚未确定。 */ + RETURNING, + + /** 库存已经幂等返还。 */ + RETURNED, + + /** 可用库存不足,未执行库存扣减。 */ + INSUFFICIENT, + + /** 库存操作超时或异常,需要查询库存流水确认结果。 */ + UNKNOWN +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/OrderEvent.java b/architecture/src/main/java/com/arch/policy/order/model/OrderEvent.java new file mode 100644 index 0000000..dc14615 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/OrderEvent.java @@ -0,0 +1,22 @@ +package com.arch.policy.order.model; + +/** 驱动订单业务状态迁移的业务事件。 */ +public enum OrderEvent { + /** 库存和三方创建流程均已成功,订单进入待支付。 */ + CREATE_SUCCEEDED, + + /** 库存不足或三方创建明确失败,订单创建失败。 */ + CREATE_FAILED, + + /** 支付成功,订单开始执行供应商预订。 */ + PAY_SUCCEEDED, + + /** 支付明确失败。 */ + PAY_FAILED, + + /** 供应商明确返回预订成功。 */ + BOOK_SUCCEEDED, + + /** 供应商明确返回预订失败。 */ + BOOK_FAILED +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/OrderRequest.java b/architecture/src/main/java/com/arch/policy/order/model/OrderRequest.java new file mode 100644 index 0000000..5107143 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/OrderRequest.java @@ -0,0 +1,65 @@ +package com.arch.policy.order.model; + +import java.io.Serializable; + +/** + * @Author : haiyang.luo + * @Date : 2026/7/22 15:46 + * @Description : + */ +public class OrderRequest implements Serializable { + + private static final long serialVersionUID = 6809741055680415850L; + + /** + * 主订单 + */ + private String orderSerialNo; + + /** + * 供应商ID + */ + private String supplierId; + + /** + * 商品ID + */ + private String skuId; + + /** + * 下单数量 + */ + private int quantity; + + public String getOrderSerialNo() { + return orderSerialNo; + } + + public void setOrderSerialNo(String orderSerialNo) { + this.orderSerialNo = orderSerialNo; + } + + public String getSupplierId() { + return supplierId; + } + + public void setSupplierId(String supplierId) { + this.supplierId = supplierId; + } + + public String getSkuId() { + return skuId; + } + + public void setSkuId(String skuId) { + this.skuId = skuId; + } + + public int getQuantity() { + return quantity; + } + + public void setQuantity(int quantity) { + this.quantity = quantity; + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/OrderResponse.java b/architecture/src/main/java/com/arch/policy/order/model/OrderResponse.java new file mode 100644 index 0000000..d4972da --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/OrderResponse.java @@ -0,0 +1,30 @@ +package com.arch.policy.order.model; + +import java.io.Serializable; + +/** + * @Author : haiyang.luo + * @Date : 2026/7/22 15:45 + * @Description : + */ +public class OrderResponse implements Serializable { + + private static final long serialVersionUID = -655337401948649968L; + + private String orderSerialNo; + private String tradeOrderSerialNo; + private String thirdPartyOrderNo; + private String status; + private String message; + + public String getOrderSerialNo() { return orderSerialNo; } + public void setOrderSerialNo(String orderSerialNo) { this.orderSerialNo = orderSerialNo; } + public String getTradeOrderSerialNo() { return tradeOrderSerialNo; } + public void setTradeOrderSerialNo(String tradeOrderSerialNo) { this.tradeOrderSerialNo = tradeOrderSerialNo; } + public String getThirdPartyOrderNo() { return thirdPartyOrderNo; } + public void setThirdPartyOrderNo(String thirdPartyOrderNo) { this.thirdPartyOrderNo = thirdPartyOrderNo; } + public String getStatus() { return status; } + public void setStatus(String status) { this.status = status; } + public String getMessage() { return message; } + public void setMessage(String message) { this.message = message; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/OrderStateMachine.java b/architecture/src/main/java/com/arch/policy/order/model/OrderStateMachine.java new file mode 100644 index 0000000..63a51ea --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/OrderStateMachine.java @@ -0,0 +1,53 @@ +package com.arch.policy.order.model; + +import org.springframework.stereotype.Component; + +/** 集中维护订单业务状态迁移规则,禁止应用服务直接指定目标状态。 */ +@Component +public final class OrderStateMachine { + + public OrderStatus fire(TradeOrder order, OrderEvent event) { + if (order == null) throw new IllegalArgumentException("order is required"); + if (event == null) throw new IllegalArgumentException("order event is required"); + + OrderStatus current = order.getStatus(); + OrderStatus target = target(current, event); + order.applyStatus(target); + return target; + } + + private static OrderStatus target(OrderStatus current, OrderEvent event) { + // Saga和消息可能至少投递一次;相同事件到达目标状态后按幂等成功处理。 + if (current == targetOf(event)) return current; + + switch (current) { + case CREATE: + if (event == OrderEvent.CREATE_SUCCEEDED) return OrderStatus.WAIT_PAY; + if (event == OrderEvent.CREATE_FAILED) return OrderStatus.CREATE_FAIL; + break; + case WAIT_PAY: + if (event == OrderEvent.PAY_SUCCEEDED) return OrderStatus.BOOKING; + if (event == OrderEvent.PAY_FAILED) return OrderStatus.PAY_FAIL; + break; + case BOOKING: + if (event == OrderEvent.BOOK_SUCCEEDED) return OrderStatus.BOOKED; + if (event == OrderEvent.BOOK_FAILED) return OrderStatus.BOOK_FAIL; + break; + default: + break; + } + throw new IllegalStateException("invalid order transition: " + current + " + " + event); + } + + private static OrderStatus targetOf(OrderEvent event) { + switch (event) { + case CREATE_SUCCEEDED: return OrderStatus.WAIT_PAY; + case CREATE_FAILED: return OrderStatus.CREATE_FAIL; + case PAY_SUCCEEDED: return OrderStatus.BOOKING; + case PAY_FAILED: return OrderStatus.PAY_FAIL; + case BOOK_SUCCEEDED: return OrderStatus.BOOKED; + case BOOK_FAILED: return OrderStatus.BOOK_FAIL; + default: throw new IllegalArgumentException("unsupported order event: " + event); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/OrderStatus.java b/architecture/src/main/java/com/arch/policy/order/model/OrderStatus.java new file mode 100644 index 0000000..60716ca --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/OrderStatus.java @@ -0,0 +1,25 @@ +package com.arch.policy.order.model; + +/** 只表达订单业务生命周期,不承载库存、三方或 Saga 技术状态。 */ +public enum OrderStatus { + /** 本地订单已经创建,创建流程尚未得出最终业务结果。 */ + CREATE, + + /** 订单创建成功,正在等待用户完成支付。 */ + WAIT_PAY, + + /** 支付完成,正在向供应商执行预订。 */ + BOOKING, + + /** 供应商已经明确返回预订成功。 */ + BOOKED, + + /** 供应商明确返回预订失败。 */ + BOOK_FAIL, + + /** 订单支付失败。 */ + PAY_FAIL, + + /** 订单创建失败,例如库存不足或三方创建明确失败。 */ + CREATE_FAIL +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/SupplierOrder.java b/architecture/src/main/java/com/arch/policy/order/model/SupplierOrder.java new file mode 100644 index 0000000..a282589 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/SupplierOrder.java @@ -0,0 +1,26 @@ +package com.arch.policy.order.model; + +/** 订单库中的供应商订单关联记录,与 TradeOrder 通过子订单号关联。 */ +public class SupplierOrder { + private final String tradeOrderSerialNo; + private final String supplierId; + private final String requestNo; + private String thirdPartyOrderNo; + private SupplierOrderStatus status; + + public SupplierOrder(String tradeOrderSerialNo, String supplierId, String requestNo, + SupplierOrderStatus status) { + this.tradeOrderSerialNo = tradeOrderSerialNo; + this.supplierId = supplierId; + this.requestNo = requestNo; + this.status = status; + } + + public String getTradeOrderSerialNo() { return tradeOrderSerialNo; } + public String getSupplierId() { return supplierId; } + public String getRequestNo() { return requestNo; } + public String getThirdPartyOrderNo() { return thirdPartyOrderNo; } + public void setThirdPartyOrderNo(String thirdPartyOrderNo) { this.thirdPartyOrderNo = thirdPartyOrderNo; } + public SupplierOrderStatus getStatus() { return status; } + public void setStatus(SupplierOrderStatus status) { this.status = status; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/SupplierOrderStatus.java b/architecture/src/main/java/com/arch/policy/order/model/SupplierOrderStatus.java new file mode 100644 index 0000000..697e9ba --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/SupplierOrderStatus.java @@ -0,0 +1,28 @@ +package com.arch.policy.order.model; + +/** 本地记录的供应商订单协作状态。 */ +public enum SupplierOrderStatus { + /** 供应商订单关联记录已经创建,尚未调用三方接口。 */ + INIT, + + /** 正在使用本地子订单号作为幂等号调用三方创建接口。 */ + CREATING, + + /** 三方已经受理请求,但订单仍处于处理中。 */ + PENDING, + + /** 三方已经明确确认订单创建成功。 */ + SUCCESS, + + /** 三方已经明确确认订单创建失败。 */ + FAILED, + + /** 调用超时或响应不明确,需要主动查询三方最终状态。 */ + UNKNOWN, + + /** 正在调用三方订单取消接口。 */ + CANCELING, + + /** 三方已经明确确认订单取消成功。 */ + CANCELED +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderResult.java b/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderResult.java new file mode 100644 index 0000000..c861e4d --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderResult.java @@ -0,0 +1,14 @@ +package com.arch.policy.order.model; + +public class ThirdOrderResult { + private final String thirdPartyOrderNo; + private final ThirdOrderStatus status; + + public ThirdOrderResult(String thirdPartyOrderNo, ThirdOrderStatus status) { + this.thirdPartyOrderNo = thirdPartyOrderNo; + this.status = status; + } + + public String getThirdPartyOrderNo() { return thirdPartyOrderNo; } + public ThirdOrderStatus getStatus() { return status; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderStatus.java b/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderStatus.java new file mode 100644 index 0000000..19e4275 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/ThirdOrderStatus.java @@ -0,0 +1,13 @@ +package com.arch.policy.order.model; + +/** 三方订单结果;超时和处理中统一按 UNKNOWN 处理,不能直接返还库存。 */ +public enum ThirdOrderStatus { + /** 三方接口明确返回业务处理成功。 */ + SUCCESS, + + /** 三方接口明确返回业务处理失败,且不会转为成功。 */ + FAILED, + + /** 三方处理中、调用超时或响应无法确认最终业务结果。 */ + UNKNOWN +} diff --git a/architecture/src/main/java/com/arch/policy/order/model/TradeOrder.java b/architecture/src/main/java/com/arch/policy/order/model/TradeOrder.java new file mode 100644 index 0000000..405a5ba --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/model/TradeOrder.java @@ -0,0 +1,25 @@ +package com.arch.policy.order.model; + +/** 本地子订单的最小快照。 */ +public class TradeOrder { + private final String orderSerialNo; + private final String tradeOrderSerialNo; + private final String supplierId; + private OrderStatus status; + + public TradeOrder(String orderSerialNo, String tradeOrderSerialNo, String supplierId, + OrderStatus status) { + this.orderSerialNo = orderSerialNo; + this.tradeOrderSerialNo = tradeOrderSerialNo; + this.supplierId = supplierId; + this.status = status; + } + + public String getOrderSerialNo() { return orderSerialNo; } + public String getTradeOrderSerialNo() { return tradeOrderSerialNo; } + public String getSupplierId() { return supplierId; } + public OrderStatus getStatus() { return status; } + + /** 仅供订单状态机应用已经校验通过的目标状态。 */ + void applyStatus(OrderStatus status) { this.status = status; } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/InventoryTransactionService.java b/architecture/src/main/java/com/arch/policy/order/service/InventoryTransactionService.java new file mode 100644 index 0000000..7143c9b --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/InventoryTransactionService.java @@ -0,0 +1,80 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.InventoryTransaction; +import com.arch.policy.order.model.InventoryTransactionStatus; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; + +/** 只负责库存库事务:正常扣库存并记录幂等库存流水,不使用预占库存。 */ +@Service +public class InventoryTransactionService { + private static final Logger LOGGER = LoggerFactory.getLogger(InventoryTransactionService.class); + private final ConcurrentMap transactions = + new ConcurrentHashMap(); + + public boolean deduct(String tradeOrderSerialNo, OrderRequest request) { + LOGGER.info("[库存库事务-开始] 查询库存流水, tradeOrderSerialNo={}, skuId={}", + tradeOrderSerialNo, request.getSkuId()); + LOGGER.info("[库存库] 原子扣减库存: stock = stock - {}, 条件 stock >= {}, skuId={}", + request.getQuantity(), request.getQuantity(), request.getSkuId()); + InventoryTransaction created = new InventoryTransaction(tradeOrderSerialNo, + request.getSkuId(), request.getQuantity(), InventoryTransactionStatus.DEDUCTED); + InventoryTransaction existing = transactions.putIfAbsent(tradeOrderSerialNo, created); + InventoryTransaction transaction = existing == null ? created : existing; + LOGGER.info("[库存库] 幂等写库存流水, tradeOrderSerialNo={}, status={}", + tradeOrderSerialNo, transaction.getStatus()); + LOGGER.info("[库存库事务-提交] 库存扣减和DEDUCTED流水同时提交"); + return transaction.getStatus() == InventoryTransactionStatus.DEDUCTED + || transaction.getStatus() == InventoryTransactionStatus.CONFIRMED; + } + + public void confirm(String tradeOrderSerialNo) { + InventoryTransaction transaction = required(tradeOrderSerialNo); + synchronized (transaction) { + if (transaction.getStatus() == InventoryTransactionStatus.DEDUCTED) { + transaction.setStatus(InventoryTransactionStatus.CONFIRMED); + } else if (transaction.getStatus() != InventoryTransactionStatus.CONFIRMED) { + throw new IllegalStateException("inventory cannot be confirmed from status: " + + transaction.getStatus()); + } + } + LOGGER.info("[库存库事务] 库存流水 {} -> {}, tradeOrderSerialNo={}", + InventoryTransactionStatus.DEDUCTED, InventoryTransactionStatus.CONFIRMED, + tradeOrderSerialNo); + } + + public void returnStock(String tradeOrderSerialNo) { + InventoryTransaction transaction = required(tradeOrderSerialNo); + synchronized (transaction) { + if (transaction.getStatus() == InventoryTransactionStatus.DEDUCTED) { + transaction.setStatus(InventoryTransactionStatus.RETURNED); + } else if (transaction.getStatus() != InventoryTransactionStatus.RETURNED) { + throw new IllegalStateException("inventory cannot be returned from status: " + + transaction.getStatus()); + } + } + LOGGER.info("[库存库事务-开始] 库存流水仅允许 {} -> {}, tradeOrderSerialNo={}", + InventoryTransactionStatus.DEDUCTED, InventoryTransactionStatus.RETURNED, + tradeOrderSerialNo); + LOGGER.info("[库存库] 只有流水状态更新成功才执行 stock = stock + quantity,防止重复返还"); + LOGGER.info("[库存库事务-提交] 库存返还和RETURNED流水同时提交"); + } + + public InventoryTransaction findByTradeOrderSerialNo(String tradeOrderSerialNo) { + return transactions.get(tradeOrderSerialNo); + } + + private InventoryTransaction required(String tradeOrderSerialNo) { + InventoryTransaction transaction = transactions.get(tradeOrderSerialNo); + if (transaction == null) { + throw new IllegalStateException("inventory transaction not found: " + + tradeOrderSerialNo); + } + return transaction; + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSaga.java b/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSaga.java new file mode 100644 index 0000000..8721d4d --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSaga.java @@ -0,0 +1,9 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.TradeOrder; + +/** 本地订单提交后的跨库、跨三方创建流程。 */ +public interface OrderCreationSaga { + void start(TradeOrder order, OrderRequest request); +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSagaStateServices.java b/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSagaStateServices.java new file mode 100644 index 0000000..8a41aeb --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/OrderCreationSagaStateServices.java @@ -0,0 +1,74 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.OrderEvent; +import com.arch.policy.order.model.SupplierOrderStatus; +import com.arch.policy.order.model.ThirdOrderResult; +import com.arch.policy.order.model.ThirdOrderStatus; +import com.arch.policy.order.model.TradeOrder; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; + +/** Seata 状态节点调用的短事务服务;未知三方结果不会错误返还库存。 */ +@Service("orderCreationSagaStateServices") +public final class OrderCreationSagaStateServices { + @Resource + private OrderTransactionService orderTransactionService; + @Resource + private InventoryTransactionService inventoryTransactionService; + @Resource + private SupplierOrderService supplierOrderService; + @Resource + private SupplierOrderTransactionService supplierOrderTransactionService; + @Resource + private OrderRecoveryService orderRecoveryService; + + public boolean deductStock(TradeOrder order, OrderRequest request) { + boolean deducted = inventoryTransactionService.deduct( + order.getTradeOrderSerialNo(), request); + return deducted; + } + + public ThirdOrderResult createThirdOrder(TradeOrder order, OrderRequest request) { + supplierOrderTransactionService.markCreating(order); + ThirdOrderResult result; + try { + result = supplierOrderService.createOrder(order.getTradeOrderSerialNo(), request); + } catch (RuntimeException uncertainFailure) { + // 网络异常只代表结果未知,不能当作三方明确失败并返还库存。 + result = new ThirdOrderResult(null, ThirdOrderStatus.UNKNOWN); + } + supplierOrderTransactionService.saveResult(order.getTradeOrderSerialNo(), + result.getThirdPartyOrderNo(), statusOf(result.getStatus())); + return result; + } + + public boolean completeSuccess(TradeOrder order) { + inventoryTransactionService.confirm(order.getTradeOrderSerialNo()); + orderTransactionService.fireEvent(order, OrderEvent.CREATE_SUCCEEDED); + return true; + } + + public boolean completeFailure(TradeOrder order) { + inventoryTransactionService.returnStock(order.getTradeOrderSerialNo()); + orderTransactionService.fireEvent(order, OrderEvent.CREATE_FAILED); + return true; + } + + public boolean completeStockFailure(TradeOrder order) { + orderTransactionService.fireEvent(order, OrderEvent.CREATE_FAILED); + return true; + } + + public boolean waitForThirdResult(TradeOrder order) { + orderRecoveryService.createThirdOrderQueryTask(order); + return true; + } + + private static SupplierOrderStatus statusOf(ThirdOrderStatus status) { + if (status == ThirdOrderStatus.SUCCESS) return SupplierOrderStatus.SUCCESS; + if (status == ThirdOrderStatus.FAILED) return SupplierOrderStatus.FAILED; + return SupplierOrderStatus.UNKNOWN; + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/OrderRecoveryService.java b/architecture/src/main/java/com/arch/policy/order/service/OrderRecoveryService.java new file mode 100644 index 0000000..3dfe8eb --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/OrderRecoveryService.java @@ -0,0 +1,67 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderEvent; +import com.arch.policy.order.model.SupplierOrder; +import com.arch.policy.order.model.SupplierOrderStatus; +import com.arch.policy.order.model.ThirdOrderResult; +import com.arch.policy.order.model.ThirdOrderStatus; +import com.arch.policy.order.model.TradeOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; + +/** 对三方超时、处理中等未知结果登记查询恢复任务。 */ +@Service +public class OrderRecoveryService { + private static final Logger LOGGER = LoggerFactory.getLogger(OrderRecoveryService.class); + + @Resource + private SupplierOrderService supplierOrderService; + @Resource + private InventoryTransactionService inventoryTransactionService; + @Resource + private OrderTransactionService orderTransactionService; + @Resource + private SupplierOrderTransactionService supplierOrderTransactionService; + + public void createThirdOrderQueryTask(TradeOrder order) { + SupplierOrder supplierOrder = supplierOrderTransactionService.findByTradeOrderSerialNo( + order.getTradeOrderSerialNo()); + LOGGER.info("[订单库事务] 创建三方订单查询任务,保持库存已扣状态,不立即返还, " + + "tradeOrderSerialNo={}, thirdPartyOrderNo={}", + order.getTradeOrderSerialNo(), thirdPartyOrderNo(supplierOrder)); + } + + /** 定时任务消费查询任务时执行;这里保留成普通方法突出恢复流程。 */ + public void reconcileThirdOrder(TradeOrder order) { + SupplierOrder supplierOrder = supplierOrderTransactionService.findByTradeOrderSerialNo( + order.getTradeOrderSerialNo()); + ThirdOrderResult result = supplierOrderService.queryOrder(order.getTradeOrderSerialNo(), + thirdPartyOrderNo(supplierOrder), order.getSupplierId()); + + if (result.getStatus() == ThirdOrderStatus.SUCCESS) { + supplierOrderTransactionService.saveResult(order.getTradeOrderSerialNo(), + result.getThirdPartyOrderNo(), SupplierOrderStatus.SUCCESS); + inventoryTransactionService.confirm(order.getTradeOrderSerialNo()); + orderTransactionService.fireEvent(order, OrderEvent.CREATE_SUCCEEDED); + return; + } + + if (result.getStatus() == ThirdOrderStatus.FAILED) { + supplierOrderTransactionService.saveResult(order.getTradeOrderSerialNo(), + result.getThirdPartyOrderNo(), SupplierOrderStatus.FAILED); + inventoryTransactionService.returnStock(order.getTradeOrderSerialNo()); + orderTransactionService.fireEvent(order, OrderEvent.CREATE_FAILED); + return; + } + + LOGGER.info("[恢复任务] 三方状态仍未知,保留已扣库存并等待下次查询, tradeOrderSerialNo={}", + order.getTradeOrderSerialNo()); + } + + private static String thirdPartyOrderNo(SupplierOrder supplierOrder) { + return supplierOrder == null ? null : supplierOrder.getThirdPartyOrderNo(); + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/OrderRpcServiceImpl.java b/architecture/src/main/java/com/arch/policy/order/service/OrderRpcServiceImpl.java new file mode 100644 index 0000000..47d40f0 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/OrderRpcServiceImpl.java @@ -0,0 +1,116 @@ +package com.arch.policy.order.service; + +import com.arch.policy.api.order.OrderRpcService; +import com.arch.policy.order.cache.OrderCacheUtil; +import com.arch.policy.order.cache.OrderLockUtil; +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.OrderResponse; +import com.arch.policy.order.model.OrderStatus; +import com.arch.policy.order.model.SupplierOrder; +import com.arch.policy.order.model.TradeOrder; +import org.apache.dubbo.config.annotation.DubboService; + +import javax.annotation.Resource; + +/** 创建本地子订单,并在本地事务提交后同步启动 Seata Saga。 */ +@DubboService(version = "1.0.0", timeout = 3000) +public class OrderRpcServiceImpl implements OrderRpcService { + @Resource + private OrderCacheUtil orderCacheUtil; + @Resource + private OrderLockUtil orderLockUtil; + @Resource + private OrderTransactionService orderTransactionService; + @Resource + private OrderCreationSaga orderCreationSaga; + @Resource + private SupplierOrderTransactionService supplierOrderTransactionService; + + @Override + public OrderResponse createOrder(OrderRequest request) { + validate(request); + String businessKey = businessKey(request); + + OrderResponse cached = orderCacheUtil.getCache(businessKey); + if (cached != null) return cached; + + if (!orderLockUtil.tryLock(businessKey)) { + return processing(request, "相同下单请求正在处理中"); + } + + try { + cached = orderCacheUtil.getCache(businessKey); + if (cached != null) return cached; + + TradeOrder order = orderTransactionService.findByRequest( + request.getOrderSerialNo(), request.getSupplierId()); + if (order != null) { + return cacheAndReturn(businessKey, order, "返回已存在的幂等订单"); + } else { + // 这里只提交订单库本地事务,不写 Outbox,也不在事务中调用库存或三方。 + order = orderTransactionService.createOrder(request); + } + + // 对同一幂等请求持锁启动,避免多个线程同时创建相同 businessKey 的 Saga。 + orderCreationSaga.start(order, request); + return cacheAndReturn(businessKey, order, message(order.getStatus())); + } finally { + orderLockUtil.unlock(businessKey); + } + } + + private OrderResponse cacheAndReturn(String businessKey, TradeOrder order, String message) { + OrderResponse result = response(order, message); + orderCacheUtil.setCache(businessKey, result); + return result; + } + + private static String message(OrderStatus status) { + if (status == OrderStatus.WAIT_PAY) return "下单成功,等待支付"; + if (status == OrderStatus.CREATE_FAIL) return "下单失败,库存已正确处理"; + return "Saga已启动,订单处理中"; + } + + private static OrderResponse processing(OrderRequest request, String message) { + OrderResponse response = new OrderResponse(); + response.setOrderSerialNo(request.getOrderSerialNo()); + response.setStatus(OrderStatus.CREATE.name()); + response.setMessage(message); + return response; + } + + private OrderResponse response(TradeOrder order, String message) { + SupplierOrder supplierOrder = supplierOrderTransactionService + .findByTradeOrderSerialNo(order.getTradeOrderSerialNo()); + OrderResponse response = new OrderResponse(); + response.setOrderSerialNo(order.getOrderSerialNo()); + response.setTradeOrderSerialNo(order.getTradeOrderSerialNo()); + response.setThirdPartyOrderNo(supplierOrder == null + ? null : supplierOrder.getThirdPartyOrderNo()); + response.setStatus(order.getStatus().name()); + response.setMessage(message); + return response; + } + + private static String businessKey(OrderRequest request) { + return request.getOrderSerialNo() + ":" + request.getSupplierId(); + } + + private static void validate(OrderRequest request) { + if (request == null) throw new IllegalArgumentException("orderRequest is required"); + if (isBlank(request.getOrderSerialNo())) { + throw new IllegalArgumentException("orderSerialNo is required"); + } + if (isBlank(request.getSupplierId())) { + throw new IllegalArgumentException("supplierId is required"); + } + if (isBlank(request.getSkuId())) throw new IllegalArgumentException("skuId is required"); + if (request.getQuantity() <= 0) { + throw new IllegalArgumentException("quantity must be positive"); + } + } + + private static boolean isBlank(String value) { + return value == null || value.trim().isEmpty(); + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/OrderTransactionService.java b/architecture/src/main/java/com/arch/policy/order/service/OrderTransactionService.java new file mode 100644 index 0000000..9948092 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/OrderTransactionService.java @@ -0,0 +1,49 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.OrderEvent; +import com.arch.policy.order.model.OrderStateMachine; +import com.arch.policy.order.model.OrderStatus; +import com.arch.policy.order.model.TradeOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; +import java.util.UUID; + +/** 只负责订单库的本地事务,示例使用日志代替真实DB操作。 */ +@Service +public class OrderTransactionService { + private static final Logger LOGGER = LoggerFactory.getLogger(OrderTransactionService.class); + @Resource + private OrderStateMachine orderStateMachine; + + public TradeOrder findByRequest(String orderSerialNo, String supplierId) { + LOGGER.info("[订单库] 按 orderSerialNo + supplierId 查询幂等订单, orderSerialNo={}, supplierId={}", + orderSerialNo, supplierId); + return null; + } + + public TradeOrder createOrder(OrderRequest request) { + String tradeOrderSerialNo = newTradeOrderSerialNo(); + LOGGER.info("[订单库事务-开始] 创建子订单"); + LOGGER.info("[订单库] 插入子订单, orderSerialNo={}, tradeOrderSerialNo={}, supplierId={}, status={}", + request.getOrderSerialNo(), tradeOrderSerialNo, request.getSupplierId(), OrderStatus.CREATE); + LOGGER.info("[订单库事务-提交] 子订单提交;提交后由应用服务同步启动Seata Saga"); + return new TradeOrder(request.getOrderSerialNo(), tradeOrderSerialNo, + request.getSupplierId(), OrderStatus.CREATE); + } + + public void fireEvent(TradeOrder order, OrderEvent event) { + OrderStatus source = order.getStatus(); + OrderStatus target = orderStateMachine.fire(order, event); + LOGGER.info("[订单库事务] 状态机迁移并CAS更新订单, tradeOrderSerialNo={}, event={}, from={}, to={}", + order.getTradeOrderSerialNo(), event, source, target); + } + + private static String newTradeOrderSerialNo() { + return "TO" + UUID.randomUUID().toString().replace("-", "") + .substring(0, 20).toUpperCase(); + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/SeataOrderCreationSaga.java b/architecture/src/main/java/com/arch/policy/order/service/SeataOrderCreationSaga.java new file mode 100644 index 0000000..c2f3858 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/SeataOrderCreationSaga.java @@ -0,0 +1,31 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.TradeOrder; +import io.seata.saga.engine.StateMachineEngine; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Component; + +import javax.annotation.Resource; +import java.util.HashMap; +import java.util.Map; + +/** 使用 tradeOrderSerialNo 作为业务幂等键启动持久化 Seata Saga。 */ +@Component +public final class SeataOrderCreationSaga implements OrderCreationSaga { + static final String STATE_MACHINE_NAME = "TradeOrderCreationSaga"; + + @Resource + private StateMachineEngine stateMachineEngine; + @Value("${order.seata.tenant-id:order}") + private String tenantId; + + @Override + public void start(TradeOrder order, OrderRequest request) { + Map context = new HashMap(); + context.put("order", order); + context.put("request", request); + stateMachineEngine.startWithBusinessKey(STATE_MACHINE_NAME, tenantId, + order.getTradeOrderSerialNo(), context); + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderService.java b/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderService.java new file mode 100644 index 0000000..81b864e --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderService.java @@ -0,0 +1,29 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.OrderRequest; +import com.arch.policy.order.model.ThirdOrderResult; +import com.arch.policy.order.model.ThirdOrderStatus; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +/** 三方订单适配器伪实现;tradeOrderSerialNo 是三方请求的稳定幂等号。 */ +@Service +public class SupplierOrderService { + private static final Logger LOGGER = LoggerFactory.getLogger(SupplierOrderService.class); + + public ThirdOrderResult createOrder(String tradeOrderSerialNo, OrderRequest request) { + LOGGER.info("[三方接口] 幂等创建订单, tradeOrderSerialNo={}, supplierId={}", + tradeOrderSerialNo, request.getSupplierId()); + // 伪代码默认模拟明确成功;真实适配器需要映射 SUCCESS、FAILED、UNKNOWN 三种结果。 + return new ThirdOrderResult("TP" + tradeOrderSerialNo, ThirdOrderStatus.SUCCESS); + } + + public ThirdOrderResult queryOrder(String tradeOrderSerialNo, String thirdPartyOrderNo, + String supplierId) { + LOGGER.info("[三方查询接口] 查询最终订单状态, tradeOrderSerialNo={}, thirdPartyOrderNo={}, supplierId={}", + tradeOrderSerialNo, thirdPartyOrderNo, supplierId); + // 伪代码默认模拟查询后成功;长期UNKNOWN需要重试,超过阈值转人工处理。 + return new ThirdOrderResult(thirdPartyOrderNo, ThirdOrderStatus.SUCCESS); + } +} diff --git a/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderTransactionService.java b/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderTransactionService.java new file mode 100644 index 0000000..dd1ad7c --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/order/service/SupplierOrderTransactionService.java @@ -0,0 +1,52 @@ +package com.arch.policy.order.service; + +import com.arch.policy.order.model.SupplierOrder; +import com.arch.policy.order.model.SupplierOrderStatus; +import com.arch.policy.order.model.TradeOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; + +/** 只负责订单库中的供应商订单关联记录;生产实现应替换为数据库事务。 */ +@Service +public class SupplierOrderTransactionService { + private static final Logger LOGGER = LoggerFactory.getLogger(SupplierOrderTransactionService.class); + private final ConcurrentMap orders = + new ConcurrentHashMap(); + + public SupplierOrder markCreating(TradeOrder order) { + SupplierOrder created = new SupplierOrder(order.getTradeOrderSerialNo(), + order.getSupplierId(), order.getTradeOrderSerialNo(), SupplierOrderStatus.CREATING); + SupplierOrder existing = orders.putIfAbsent(order.getTradeOrderSerialNo(), created); + SupplierOrder supplierOrder = existing == null ? created : existing; + LOGGER.info("[订单库事务] 幂等创建供应商订单记录, tradeOrderSerialNo={}, status={}", + order.getTradeOrderSerialNo(), supplierOrder.getStatus()); + return supplierOrder; + } + + public void saveResult(String tradeOrderSerialNo, String thirdPartyOrderNo, + SupplierOrderStatus status) { + SupplierOrder supplierOrder = required(tradeOrderSerialNo); + synchronized (supplierOrder) { + supplierOrder.setThirdPartyOrderNo(thirdPartyOrderNo); + supplierOrder.setStatus(status); + } + LOGGER.info("[订单库事务] 保存供应商订单结果, tradeOrderSerialNo={}, thirdPartyOrderNo={}, status={}", + tradeOrderSerialNo, thirdPartyOrderNo, status); + } + + public SupplierOrder findByTradeOrderSerialNo(String tradeOrderSerialNo) { + return orders.get(tradeOrderSerialNo); + } + + private SupplierOrder required(String tradeOrderSerialNo) { + SupplierOrder supplierOrder = orders.get(tradeOrderSerialNo); + if (supplierOrder == null) { + throw new IllegalStateException("supplier order not found: " + tradeOrderSerialNo); + } + return supplierOrder; + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/AsyncSearchCoordinator.java b/architecture/src/main/java/com/arch/policy/search/application/AsyncSearchCoordinator.java new file mode 100644 index 0000000..6491af2 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/AsyncSearchCoordinator.java @@ -0,0 +1,76 @@ +package com.arch.policy.search.application; + +import com.arch.policy.common.search.PolicySearchRequest; +import com.arch.policy.common.search.PolicySearchResponse; +import com.arch.policy.common.search.SupplierCallbackRequest; +import com.fasterxml.jackson.databind.ObjectMapper; + +import java.util.ArrayList; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; + +public final class AsyncSearchCoordinator { + private static final long MAX_WAIT_SLICE_MILLIS = 200; + private final SearchStateStore stateStore; + private final SupplierTaskDispatcher dispatcher; + private final LocalSearchWaiters localWaiters; + private final ObjectMapper objectMapper; + private final long redisTtlSeconds; + + public AsyncSearchCoordinator(SearchStateStore stateStore, SupplierTaskDispatcher dispatcher, + LocalSearchWaiters localWaiters, ObjectMapper objectMapper, + long redisTtlSeconds) { + this.stateStore = stateStore; + this.dispatcher = dispatcher; + this.localWaiters = localWaiters; + this.objectMapper = objectMapper; + this.redisTtlSeconds = redisTtlSeconds; + } + + public PolicySearchResponse search(PolicySearchRequest request) throws Exception { + validate(request); + String searchKey = UUID.randomUUID().toString(); + Set suppliers = new LinkedHashSet(request.getSupplierIds()); + stateStore.initialize(searchKey, suppliers, redisTtlSeconds); + LocalSearchWaiters.Waiter waiter = localWaiters.register(searchKey); + try { + if (stateStore.getState(searchKey) == SearchState.COMPLETED) return aggregate(searchKey); + dispatcher.dispatch(searchKey, suppliers, request); + return awaitResults(searchKey, request.getTotalTimeoutMillis(), waiter); + } finally { + localWaiters.remove(searchKey, waiter); + } + } + + private PolicySearchResponse awaitResults(String searchKey, long timeoutMillis, + LocalSearchWaiters.Waiter waiter) throws Exception { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (true) { + SearchState state = stateStore.getState(searchKey); + if (state == SearchState.COMPLETED) return aggregate(searchKey); + long remaining = deadline - System.currentTimeMillis(); + if (state == SearchState.TIMED_OUT || remaining <= 0) { + stateStore.markTimedOut(searchKey, redisTtlSeconds); + return aggregate(searchKey); + } + waiter.await(Math.min(remaining, MAX_WAIT_SLICE_MILLIS)); + } + } + + private PolicySearchResponse aggregate(String searchKey) throws Exception { + List results = new ArrayList(); + for (String payload : stateStore.getResultPayloads(searchKey)) { + results.add(objectMapper.readValue(payload, SupplierCallbackRequest.class)); + } + SearchState state = stateStore.getState(searchKey); + return new PolicySearchResponse(searchKey, state.name(), results); + } + + private static void validate(PolicySearchRequest request) { + if (request == null) throw new IllegalArgumentException("request is required"); + if (request.getTotalTimeoutMillis() <= 0) throw new IllegalArgumentException("timeout must be positive"); + if (request.getSupplierIds().isEmpty()) throw new IllegalArgumentException("supplierIds is required"); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/LocalSearchWaiters.java b/architecture/src/main/java/com/arch/policy/search/application/LocalSearchWaiters.java new file mode 100644 index 0000000..66c27c7 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/LocalSearchWaiters.java @@ -0,0 +1,32 @@ +package com.arch.policy.search.application; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +public final class LocalSearchWaiters { + private final ConcurrentMap waiters = new ConcurrentHashMap(); + + public Waiter register(String searchKey) { + Waiter waiter = new Waiter(); + Waiter existing = waiters.putIfAbsent(searchKey, waiter); + if (existing != null) throw new IllegalStateException("duplicate local search: " + searchKey); + return waiter; + } + + public void remove(String searchKey, Waiter waiter) { waiters.remove(searchKey, waiter); } + + public void signal(String searchKey) { + Waiter waiter = waiters.get(searchKey); + if (waiter != null) waiter.signal(); + } + + public static final class Waiter { + private final CountDownLatch completed = new CountDownLatch(1); + public void await(long millis) throws InterruptedException { + completed.await(millis, TimeUnit.MILLISECONDS); + } + private void signal() { completed.countDown(); } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/PolicyIncrementalUpdater.java b/architecture/src/main/java/com/arch/policy/search/application/PolicyIncrementalUpdater.java new file mode 100644 index 0000000..b8deab9 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/PolicyIncrementalUpdater.java @@ -0,0 +1,16 @@ +package com.arch.policy.search.application; + +import com.arch.policy.common.model.PolicyChange; +import com.arch.policy.search.domain.snapshot.ActiveSnapshotRegistry; + +public final class PolicyIncrementalUpdater { + private final ActiveSnapshotRegistry registry; + + public PolicyIncrementalUpdater(ActiveSnapshotRegistry registry) { this.registry = registry; } + + public boolean apply(PolicyChange change) throws Exception { + try (ActiveSnapshotRegistry.SnapshotLease lease = registry.acquire()) { + return lease.snapshot().apply(change); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/PolicyStartupRunner.java b/architecture/src/main/java/com/arch/policy/search/application/PolicyStartupRunner.java new file mode 100644 index 0000000..40ec539 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/PolicyStartupRunner.java @@ -0,0 +1,19 @@ +package com.arch.policy.search.application; + +import com.arch.policy.search.domain.snapshot.PolicySnapshotService; +import org.springframework.boot.ApplicationArguments; +import org.springframework.boot.ApplicationRunner; + +public final class PolicyStartupRunner implements ApplicationRunner { + private final PolicySnapshotService snapshotService; + private final String initialVersion; + + public PolicyStartupRunner(PolicySnapshotService snapshotService, String initialVersion) { + this.snapshotService = snapshotService; + this.initialVersion = initialVersion; + } + + @Override public void run(ApplicationArguments args) throws Exception { + snapshotService.initializeBlocking(initialVersion); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/SearchState.java b/architecture/src/main/java/com/arch/policy/search/application/SearchState.java new file mode 100644 index 0000000..140010a --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/SearchState.java @@ -0,0 +1,3 @@ +package com.arch.policy.search.application; + +public enum SearchState { WAITING, COMPLETED, TIMED_OUT } diff --git a/architecture/src/main/java/com/arch/policy/search/application/SearchStateStore.java b/architecture/src/main/java/com/arch/policy/search/application/SearchStateStore.java new file mode 100644 index 0000000..d4bff02 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/SearchStateStore.java @@ -0,0 +1,13 @@ +package com.arch.policy.search.application; + +import java.util.List; +import java.util.Set; + +public interface SearchStateStore { + void initialize(String searchKey, Set supplierIds, long ttlSeconds); + SearchState getState(String searchKey); + List getResultPayloads(String searchKey); + void recordCallback(String searchKey, String supplierId, String payload, + boolean finished, long ttlSeconds); + void markTimedOut(String searchKey, long ttlSeconds); +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/SupplierCallbackService.java b/architecture/src/main/java/com/arch/policy/search/application/SupplierCallbackService.java new file mode 100644 index 0000000..c3d8c59 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/SupplierCallbackService.java @@ -0,0 +1,38 @@ +package com.arch.policy.search.application; + +import com.arch.policy.common.search.SupplierCallbackRequest; +import com.fasterxml.jackson.databind.ObjectMapper; + +public final class SupplierCallbackService { + private final SearchStateStore stateStore; + private final ObjectMapper objectMapper; + private final long redisTtlSeconds; + + public SupplierCallbackService(SearchStateStore stateStore, ObjectMapper objectMapper, + long redisTtlSeconds) { + this.stateStore = stateStore; + this.objectMapper = objectMapper; + this.redisTtlSeconds = redisTtlSeconds; + } + + public void callback(SupplierCallbackRequest request) throws Exception { + validate(request); + String payload = request.getResults().isEmpty() ? "" : objectMapper.writeValueAsString(request); + stateStore.recordCallback(request.getSearchKey(), request.getSupplierId(), payload, + request.isSearchFinished(), redisTtlSeconds); + } + + public void markFinishedWithoutResult(String searchKey, String supplierId) throws Exception { + SupplierCallbackRequest request = new SupplierCallbackRequest(); + request.setSearchKey(searchKey); + request.setSupplierId(supplierId); + request.setSearchFinished(true); + callback(request); + } + + private static void validate(SupplierCallbackRequest request) { + if (request == null || request.getSearchKey() == null || request.getSupplierId() == null) { + throw new IllegalArgumentException("searchKey and supplierId are required"); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/application/SupplierTaskDispatcher.java b/architecture/src/main/java/com/arch/policy/search/application/SupplierTaskDispatcher.java new file mode 100644 index 0000000..213e734 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/application/SupplierTaskDispatcher.java @@ -0,0 +1,9 @@ +package com.arch.policy.search.application; + +import com.arch.policy.common.search.PolicySearchRequest; + +import java.util.Set; + +public interface SupplierTaskDispatcher { + void dispatch(String searchKey, Set supplierIds, PolicySearchRequest request); +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/ActiveSnapshotRegistry.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/ActiveSnapshotRegistry.java new file mode 100644 index 0000000..44cb02b --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/ActiveSnapshotRegistry.java @@ -0,0 +1,61 @@ +package com.arch.policy.search.domain.snapshot; + +import java.util.concurrent.atomic.AtomicBoolean; + +/** Atomic activation plus draining of queries that still hold the old version. */ +public final class ActiveSnapshotRegistry implements AutoCloseable { + private Entry active; + + public synchronized boolean isReady() { return active != null; } + + public synchronized SnapshotLease acquire() { + if (active == null) throw new IllegalStateException("policy snapshot is not ready"); + active.references++; + return new SnapshotLease(active); + } + + public synchronized void activate(PolicySnapshot snapshot) { + Entry previous = active; + active = new Entry(snapshot); + if (previous != null) retire(previous); + } + + @Override public synchronized void close() { + Entry previous = active; + active = null; + if (previous != null) retire(previous); + } + + private void release(Entry entry) { + synchronized (this) { + entry.references--; + closeWhenDrained(entry); + } + } + + private void retire(Entry entry) { + entry.retired = true; + closeWhenDrained(entry); + } + + private void closeWhenDrained(Entry entry) { + if (entry.retired && entry.references == 0) entry.snapshot.closeAndDelete(); + } + + private static final class Entry { + private final PolicySnapshot snapshot; + private int references; + private boolean retired; + private Entry(PolicySnapshot snapshot) { this.snapshot = snapshot; } + } + + public final class SnapshotLease implements AutoCloseable { + private final Entry entry; + private final AtomicBoolean released = new AtomicBoolean(); + private SnapshotLease(Entry entry) { this.entry = entry; } + public PolicySnapshot snapshot() { return entry.snapshot; } + @Override public void close() { + if (released.compareAndSet(false, true)) release(entry); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/DefaultSnapshotValidator.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/DefaultSnapshotValidator.java new file mode 100644 index 0000000..2c51ddf --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/DefaultSnapshotValidator.java @@ -0,0 +1,17 @@ +package com.arch.policy.search.domain.snapshot; + +import com.arch.policy.common.model.MessagePosition; + +import static com.arch.policy.search.domain.snapshot.SnapshotPorts.SnapshotValidator; + +/** Baseline invariants; domain-specific checks can be supplied through SnapshotValidator. */ +public final class DefaultSnapshotValidator implements SnapshotValidator { + @Override public void validate(PolicySnapshot candidate, MessagePosition expectedPosition) throws Exception { + if (!candidate.getPosition().equals(expectedPosition)) { + throw new IllegalStateException("message position mismatch"); + } + if (candidate.policyCount() != candidate.indexedPolicyCount()) { + throw new IllegalStateException("RocksDB and bitmap policy counts differ"); + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/FileSystemSnapshotDirectory.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/FileSystemSnapshotDirectory.java new file mode 100644 index 0000000..663fed2 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/FileSystemSnapshotDirectory.java @@ -0,0 +1,45 @@ +package com.arch.policy.search.domain.snapshot; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Comparator; +import java.util.stream.Stream; + +import static com.arch.policy.search.domain.snapshot.SnapshotPorts.SnapshotDirectory; + +/** Keeps every version in an isolated directory and removes abandoned candidates. */ +public final class FileSystemSnapshotDirectory implements SnapshotDirectory { + private final Path root; + + public FileSystemSnapshotDirectory(Path root) { this.root = root; } + + @Override public Path create(String version) throws IOException { + Path directory = root.resolve(safeVersion(version)); + delete(directory); + return Files.createDirectories(directory); + } + + @Override public void delete(Path directory) throws IOException { + if (!Files.exists(directory)) return; + try (Stream paths = Files.walk(directory)) { + paths.sorted(Comparator.reverseOrder()).forEach(path -> { + try { Files.deleteIfExists(path); } + catch (IOException failure) { throw new DeleteFailure(failure); } + }); + } catch (DeleteFailure failure) { + throw (IOException) failure.getCause(); + } + } + + private static String safeVersion(String version) { + if (version == null || !version.matches("[A-Za-z0-9._-]+")) { + throw new IllegalArgumentException("invalid snapshot version: " + version); + } + return version; + } + + private static final class DeleteFailure extends RuntimeException { + private DeleteFailure(IOException cause) { super(cause); } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshot.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshot.java new file mode 100644 index 0000000..eb13c1d --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshot.java @@ -0,0 +1,174 @@ +package com.arch.policy.search.domain.snapshot; + +import com.arch.policy.common.model.MessagePosition; +import com.arch.policy.common.model.PolicyChange; +import com.arch.policy.common.model.PolicyRecord; + +import org.roaringbitmap.RoaringBitmap; +import org.rocksdb.Options; +import org.rocksdb.RocksDB; +import org.rocksdb.RocksDBException; +import org.rocksdb.RocksIterator; + +import java.nio.ByteBuffer; +import java.nio.file.Files; +import java.nio.file.Path; +import java.io.IOException; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import java.util.stream.Stream; + +/** A version-isolated RocksDB detail store and its matching bitmap index. */ +public final class PolicySnapshot implements AutoCloseable { + static { RocksDB.loadLibrary(); } + + private final String version; + private final Path directory; + private final Options options; + private final RocksDB database; + private final Map bitmapIndex = new HashMap(); + private final RoaringBitmap allPolicyIds = new RoaringBitmap(); + private final Map> termsByPolicy = new HashMap>(); + private MessagePosition position = MessagePosition.BEGINNING; + private boolean closed; + + public PolicySnapshot(String version, Path directory) throws RocksDBException { + this.version = version; + this.directory = directory; + this.options = new Options().setCreateIfMissing(true); + this.database = RocksDB.open(options, directory.toString()); + } + + public synchronized void upsert(PolicyRecord policy) throws RocksDBException { + ensureOpen(); + removeFromIndex(policy.getId()); + database.put(key(policy.getId()), policy.getDetail()); + allPolicyIds.add(policy.getId()); + termsByPolicy.put(policy.getId(), policy.getIndexTerms()); + for (String term : policy.getIndexTerms()) { + RoaringBitmap bitmap = bitmapIndex.get(term); + if (bitmap == null) { + bitmap = new RoaringBitmap(); + bitmapIndex.put(term, bitmap); + } + bitmap.add(policy.getId()); + } + } + + public synchronized void delete(int policyId) throws RocksDBException { + ensureOpen(); + database.delete(key(policyId)); + allPolicyIds.remove(policyId); + removeFromIndex(policyId); + } + + public synchronized byte[] findDetail(int policyId) throws RocksDBException { + ensureOpen(); + return database.get(key(policyId)); + } + + public synchronized RoaringBitmap findPolicyIds(String term) { + ensureOpen(); + RoaringBitmap bitmap = bitmapIndex.get(term); + return bitmap == null ? new RoaringBitmap() : bitmap.clone(); + } + + public synchronized RoaringBitmap allPolicyIds() { + ensureOpen(); + return allPolicyIds.clone(); + } + + /** Applies an ordered event and advances its position in the same monitor. */ + public synchronized boolean apply(PolicyChange change) throws RocksDBException { + ensureOpen(); + if (change.getPosition().compareTo(position) <= 0) return false; + if (change.getType() == PolicyChange.Type.DELETE) delete(change.getPolicyId()); + else upsert(change.getPolicy()); + advanceTo(change.getPosition()); + return true; + } + + public synchronized long policyCount() throws RocksDBException { + ensureOpen(); + long count = 0; + try (RocksIterator iterator = database.newIterator()) { + for (iterator.seekToFirst(); iterator.isValid(); iterator.next()) count++; + iterator.status(); + } + return count; + } + + public synchronized Map bitmapCardinalities() { + ensureOpen(); + Map result = new HashMap(); + for (Map.Entry entry : bitmapIndex.entrySet()) { + result.put(entry.getKey(), entry.getValue().getCardinality()); + } + return Collections.unmodifiableMap(result); + } + + public synchronized int indexedPolicyCount() { + ensureOpen(); + RoaringBitmap all = new RoaringBitmap(); + for (RoaringBitmap bitmap : bitmapIndex.values()) all.or(bitmap); + return all.getCardinality(); + } + + public String getVersion() { return version; } + public Path getDirectory() { return directory; } + public synchronized MessagePosition getPosition() { return position; } + public synchronized void advanceTo(MessagePosition newPosition) { + ensureOpen(); + if (newPosition.compareTo(position) < 0) throw new IllegalArgumentException("position cannot move backwards"); + position = newPosition; + } + + @Override public synchronized void close() { + if (closed) return; + closed = true; + database.close(); + options.close(); + bitmapIndex.clear(); + allPolicyIds.clear(); + termsByPolicy.clear(); + } + + public void closeAndDelete() { + close(); + try { + if (!Files.exists(directory)) return; + try (Stream paths = Files.walk(directory)) { + paths.sorted(java.util.Comparator.reverseOrder()).forEach(path -> { + try { Files.deleteIfExists(path); } + catch (IOException failure) { throw new DeleteFailure(failure); } + }); + } + } catch (IOException failure) { + throw new IllegalStateException("cannot delete snapshot " + directory, failure); + } catch (DeleteFailure failure) { + throw new IllegalStateException("cannot delete snapshot " + directory, failure.getCause()); + } + } + + private static final class DeleteFailure extends RuntimeException { + private DeleteFailure(IOException cause) { super(cause); } + } + + private void removeFromIndex(int policyId) { + Set oldTerms = termsByPolicy.remove(policyId); + if (oldTerms == null) return; + for (String term : oldTerms) { + RoaringBitmap bitmap = bitmapIndex.get(term); + bitmap.remove(policyId); + if (bitmap.isEmpty()) bitmapIndex.remove(term); + } + } + + private void ensureOpen() { + if (closed) throw new IllegalStateException("snapshot is closed: " + version); + } + + private static byte[] key(int id) { return ByteBuffer.allocate(4).putInt(id).array(); } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshotService.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshotService.java new file mode 100644 index 0000000..f2f35f9 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/PolicySnapshotService.java @@ -0,0 +1,80 @@ +package com.arch.policy.search.domain.snapshot; + +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; + +/** Coordinates startup retry and side-by-side runtime refresh. */ +public final class PolicySnapshotService implements AutoCloseable { + private final SnapshotBuilder builder; + private final ActiveSnapshotRegistry registry; + private final ScheduledExecutorService executor; + private final long retryDelayMillis; + private final AtomicBoolean building = new AtomicBoolean(); + private volatile boolean stopped; + + public PolicySnapshotService(SnapshotBuilder builder, ActiveSnapshotRegistry registry, + ScheduledExecutorService executor, long retryDelayMillis) { + this.builder = builder; + this.registry = registry; + this.executor = executor; + this.retryDelayMillis = retryDelayMillis; + } + + /** Keeps readiness false and retries until the first valid snapshot is activated. */ + public void start(String version) { submitBuild(version, true); } + + /** Returns false when another candidate is already being built. */ + public boolean refresh(String version) { return submitBuild(version, false); } + + public boolean isReady() { return registry.isReady(); } + public boolean isBuilding() { return building.get(); } + + /** Used by application startup: Spring startup does not finish before a snapshot is ready. */ + public void initializeBlocking(String version) throws InterruptedException { + if (!building.compareAndSet(false, true)) throw new IllegalStateException("snapshot build already running"); + try { + while (!stopped && !registry.isReady()) { + try { + registry.activate(builder.build(version)); + } catch (Exception failure) { + Thread.sleep(retryDelayMillis); + } + } + } finally { + building.set(false); + } + } + + private boolean submitBuild(final String version, final boolean retryOnFailure) { + if (stopped || !building.compareAndSet(false, true)) return false; + executor.execute(new Runnable() { + @Override public void run() { buildAndActivate(version, retryOnFailure); } + }); + return true; + } + + private void buildAndActivate(final String version, final boolean retryOnFailure) { + try { + PolicySnapshot candidate = builder.build(version); + if (stopped) candidate.closeAndDelete(); + else registry.activate(candidate); + } catch (Exception ignored) { + if (retryOnFailure && !stopped) { + executor.schedule(new Runnable() { + @Override public void run() { buildAndActivate(version, true); } + }, retryDelayMillis, TimeUnit.MILLISECONDS); + return; + } + } finally { + // Startup retry owns the build slot until it succeeds or the service stops. + if (!retryOnFailure || registry.isReady() || stopped) building.set(false); + } + } + + @Override public void close() { + stopped = true; + registry.close(); + executor.shutdownNow(); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotBuilder.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotBuilder.java new file mode 100644 index 0000000..49b8797 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotBuilder.java @@ -0,0 +1,58 @@ +package com.arch.policy.search.domain.snapshot; + +import com.arch.policy.common.model.MessagePosition; + +import org.rocksdb.RocksDBException; + +import java.nio.file.Path; + +import static com.arch.policy.search.domain.snapshot.SnapshotPorts.*; + +/** Builds a candidate without exposing it to queries. */ +public final class SnapshotBuilder { + private final FullPolicyLoader fullLoader; + private final IncrementalReplayer replayer; + private final SnapshotValidator validator; + private final SnapshotDirectory directories; + + public SnapshotBuilder(FullPolicyLoader fullLoader, IncrementalReplayer replayer, + SnapshotValidator validator, SnapshotDirectory directories) { + this.fullLoader = fullLoader; + this.replayer = replayer; + this.validator = validator; + this.directories = directories; + } + + public PolicySnapshot build(String version) throws Exception { + Path directory = directories.create(version); + PolicySnapshot candidate = null; + try { + candidate = open(version, directory); + MessagePosition position = fullLoader.loadInto(candidate); + candidate.advanceTo(position); + catchUp(candidate); + validator.validate(candidate, candidate.getPosition()); + return candidate; + } catch (Exception failure) { + if (candidate != null) candidate.close(); + directories.delete(directory); + throw failure; + } + } + + private void catchUp(PolicySnapshot candidate) throws Exception { + while (true) { + MessagePosition target = replayer.latestPosition(); + if (candidate.getPosition().compareTo(target) >= 0) return; + MessagePosition replayed = replayer.replayInto(candidate, candidate.getPosition(), target); + if (replayed.compareTo(candidate.getPosition()) <= 0 || replayed.compareTo(target) > 0) { + throw new IllegalStateException("incremental replay made invalid progress"); + } + candidate.advanceTo(replayed); + } + } + + private PolicySnapshot open(String version, Path directory) throws RocksDBException { + return new PolicySnapshot(version, directory); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotPorts.java b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotPorts.java new file mode 100644 index 0000000..f41f262 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/domain/snapshot/SnapshotPorts.java @@ -0,0 +1,30 @@ +package com.arch.policy.search.domain.snapshot; + +import com.arch.policy.common.model.MessagePosition; + +import java.nio.file.Path; + +public final class SnapshotPorts { + private SnapshotPorts() {} + + public interface FullPolicyLoader { + /** Returns the message position corresponding to the full-data cut. */ + MessagePosition loadInto(PolicySnapshot target) throws Exception; + } + + public interface IncrementalReplayer { + /** Replays (position, latest] and returns the actual replayed position. */ + MessagePosition replayInto(PolicySnapshot target, MessagePosition position, + MessagePosition latest) throws Exception; + MessagePosition latestPosition() throws Exception; + } + + public interface SnapshotValidator { + void validate(PolicySnapshot candidate, MessagePosition expectedPosition) throws Exception; + } + + public interface SnapshotDirectory { + Path create(String version) throws Exception; + void delete(Path directory) throws Exception; + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/demo/DemoSupplierTaskDispatcher.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/demo/DemoSupplierTaskDispatcher.java new file mode 100644 index 0000000..7826bf0 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/demo/DemoSupplierTaskDispatcher.java @@ -0,0 +1,47 @@ +package com.arch.policy.search.infrastructure.demo; + +import com.arch.policy.search.application.SupplierCallbackService; +import com.arch.policy.search.application.SupplierTaskDispatcher; +import com.arch.policy.common.search.PolicySearchRequest; +import com.arch.policy.common.search.SupplierCallbackRequest; + +import java.util.Collections; +import java.util.Set; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; + +/** Simulates downstream suppliers asynchronously calling this module back. */ +public final class DemoSupplierTaskDispatcher implements SupplierTaskDispatcher { + private final ScheduledExecutorService executor; + private final SupplierCallbackService callbackService; + + public DemoSupplierTaskDispatcher(ScheduledExecutorService executor, + SupplierCallbackService callbackService) { + this.executor = executor; + this.callbackService = callbackService; + } + + @Override public void dispatch(final String searchKey, Set supplierIds, + final PolicySearchRequest request) { + int delay = 20; + for (final String supplierId : supplierIds) { + executor.schedule(new Runnable() { + @Override public void run() { + try { + SupplierCallbackRequest callback = new SupplierCallbackRequest(); + callback.setSearchKey(searchKey); + callback.setSupplierId(supplierId); + callback.setResults(Collections.singletonList( + supplierId + " quote for " + request.getCriteria())); + callback.setSearchFinished(true); + callbackService.callback(callback); + } catch (Exception failure) { + try { callbackService.markFinishedWithoutResult(searchKey, supplierId); } + catch (Exception ignored) { /* broker retry/alert belongs in a real adapter */ } + } + } + }, delay, TimeUnit.MILLISECONDS); + delay += 20; + } + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/KafkaPolicyChangeListener.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/KafkaPolicyChangeListener.java new file mode 100644 index 0000000..e935aa6 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/KafkaPolicyChangeListener.java @@ -0,0 +1,45 @@ +package com.arch.policy.search.infrastructure.kafka; + +import com.arch.policy.common.model.MessagePosition; +import com.arch.policy.common.model.PolicyChange; +import com.arch.policy.common.model.PolicyRecord; +import com.arch.policy.search.application.PolicyIncrementalUpdater; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.springframework.kafka.annotation.KafkaListener; + +import java.nio.charset.StandardCharsets; +import java.util.HashSet; + +public final class KafkaPolicyChangeListener { + private final ObjectMapper objectMapper; + private final PolicyIncrementalUpdater updater; + + public KafkaPolicyChangeListener(ObjectMapper objectMapper, PolicyIncrementalUpdater updater) { + this.objectMapper = objectMapper; + this.updater = updater; + } + + @KafkaListener(topics = "${policy.kafka.topic:policy-change}", + groupId = "${policy.kafka.group-id:policy-search}") + public void onMessage(String json) throws Exception { + PolicyChangeMessage message = objectMapper.readValue(json, PolicyChangeMessage.class); + updater.apply(toChange(message)); + } + + private PolicyChange toChange(PolicyChangeMessage message) { + MessagePosition position = new MessagePosition(message.getPosition()); + if ("DELETE".equalsIgnoreCase(message.getOperation())) { + return PolicyChange.delete(message.getPolicyId(), position); + } + if (!"UPSERT".equalsIgnoreCase(message.getOperation())) { + throw new IllegalArgumentException("unsupported policy operation: " + message.getOperation()); + } + if (message.getDetail() == null || message.getIndexTerms() == null) { + throw new IllegalArgumentException("UPSERT requires detail and indexTerms"); + } + PolicyRecord policy = new PolicyRecord(message.getPolicyId(), + message.getDetail().getBytes(StandardCharsets.UTF_8), + new HashSet(message.getIndexTerms())); + return PolicyChange.upsert(policy, position); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/PolicyChangeMessage.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/PolicyChangeMessage.java new file mode 100644 index 0000000..9718d5f --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/kafka/PolicyChangeMessage.java @@ -0,0 +1,23 @@ +package com.arch.policy.search.infrastructure.kafka; + +import java.util.ArrayList; +import java.util.List; + +public final class PolicyChangeMessage { + private String operation; + private int policyId; + private String detail; + private List indexTerms = new ArrayList(); + private long position; + + public String getOperation() { return operation; } + public void setOperation(String operation) { this.operation = operation; } + public int getPolicyId() { return policyId; } + public void setPolicyId(int policyId) { this.policyId = policyId; } + public String getDetail() { return detail; } + public void setDetail(String detail) { this.detail = detail; } + public List getIndexTerms() { return indexTerms; } + public void setIndexTerms(List indexTerms) { this.indexTerms = indexTerms; } + public long getPosition() { return position; } + public void setPosition(long position) { this.position = position; } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/RedisSearchStateStore.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/RedisSearchStateStore.java new file mode 100644 index 0000000..6347e52 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/RedisSearchStateStore.java @@ -0,0 +1,84 @@ +package com.arch.policy.search.infrastructure.redis; + +import com.arch.policy.search.application.SearchState; +import com.arch.policy.search.application.SearchStateStore; +import org.springframework.data.redis.core.StringRedisTemplate; +import org.springframework.data.redis.core.script.DefaultRedisScript; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +public final class RedisSearchStateStore implements SearchStateStore { + public static final String FINISHED_CHANNEL = "search-finished"; + private static final String EMPTY_RESULT_MARKER = "__SEARCH_INITIALIZED__"; + private static final DefaultRedisScript INIT = script( + "redis.call('DEL', KEYS[1], KEYS[2], KEYS[3]); redis.call('RPUSH', KEYS[1], ARGV[2]); " + + "redis.call('EXPIRE', KEYS[1], ARGV[1]); " + + "for i=3,#ARGV do redis.call('SADD', KEYS[2], ARGV[i]); end; " + + "if redis.call('SCARD', KEYS[2]) == 0 then " + + " redis.call('SET', KEYS[3], 'COMPLETED', 'EX', ARGV[1]); return 1; " + + "end; redis.call('SET', KEYS[3], 'WAITING', 'EX', ARGV[1]); " + + "redis.call('EXPIRE', KEYS[1], ARGV[1]); redis.call('EXPIRE', KEYS[2], ARGV[1]); return 0;"); + private static final DefaultRedisScript CALLBACK = script( + "if redis.call('GET', KEYS[3]) ~= 'WAITING' then return 0; end; " + + "if redis.call('SISMEMBER', KEYS[2], ARGV[2]) == 0 then return 0; end; " + + "if ARGV[1] ~= '' then redis.call('RPUSH', KEYS[1], ARGV[1]); end; " + + "local removed=0; if ARGV[3] == 'true' then removed=redis.call('SREM', KEYS[2], ARGV[2]); end; " + + "redis.call('EXPIRE', KEYS[1], ARGV[5]); redis.call('EXPIRE', KEYS[2], ARGV[5]); " + + "redis.call('EXPIRE', KEYS[3], ARGV[5]); " + + "if removed > 0 and redis.call('SCARD', KEYS[2]) == 0 then " + + " redis.call('SET', KEYS[3], 'COMPLETED', 'EX', ARGV[5]); " + + " redis.call('PUBLISH', ARGV[6], ARGV[4]); return 2; end; return 1;"); + private static final DefaultRedisScript TIMEOUT = script( + "if redis.call('GET', KEYS[3]) == 'WAITING' then " + + " redis.call('SET', KEYS[3], 'TIMED_OUT', 'EX', ARGV[1]); " + + " redis.call('EXPIRE', KEYS[1], ARGV[1]); redis.call('EXPIRE', KEYS[2], ARGV[1]); return 1; end; return 0;"); + + private final StringRedisTemplate redis; + + public RedisSearchStateStore(StringRedisTemplate redis) { this.redis = redis; } + + @Override public void initialize(String searchKey, Set supplierIds, long ttlSeconds) { + List args = new ArrayList(); + args.add(Long.toString(ttlSeconds)); + args.add(EMPTY_RESULT_MARKER); + args.addAll(supplierIds); + redis.execute(INIT, keys(searchKey), args.toArray()); + } + + @Override public SearchState getState(String searchKey) { + String state = redis.opsForValue().get(stateKey(searchKey)); + return state == null ? SearchState.TIMED_OUT : SearchState.valueOf(state); + } + + @Override public List getResultPayloads(String searchKey) { + List values = redis.opsForList().range(resultKey(searchKey), 0, -1); + if (values == null) return Collections.emptyList(); + List results = new ArrayList(values); + results.remove(EMPTY_RESULT_MARKER); + return results; + } + + @Override public void recordCallback(String searchKey, String supplierId, String payload, + boolean finished, long ttlSeconds) { + redis.execute(CALLBACK, keys(searchKey), payload, supplierId, Boolean.toString(finished), + searchKey, Long.toString(ttlSeconds), FINISHED_CHANNEL); + } + + @Override public void markTimedOut(String searchKey, long ttlSeconds) { + redis.execute(TIMEOUT, keys(searchKey), Long.toString(ttlSeconds)); + } + + private static List keys(String searchKey) { + return Arrays.asList(resultKey(searchKey), pendingKey(searchKey), stateKey(searchKey)); + } + private static String resultKey(String key) { return "search:" + key + ":results"; } + private static String pendingKey(String key) { return "search:" + key + ":pending"; } + private static String stateKey(String key) { return "search:" + key + ":state"; } + private static DefaultRedisScript script(String source) { + return new DefaultRedisScript(source, Long.class); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/SearchFinishedSubscriber.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/SearchFinishedSubscriber.java new file mode 100644 index 0000000..208c8f0 --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/redis/SearchFinishedSubscriber.java @@ -0,0 +1,15 @@ +package com.arch.policy.search.infrastructure.redis; + +import com.arch.policy.search.application.LocalSearchWaiters; +import org.springframework.data.redis.connection.Message; +import org.springframework.data.redis.connection.MessageListener; + +import java.nio.charset.StandardCharsets; + +public final class SearchFinishedSubscriber implements MessageListener { + private final LocalSearchWaiters waiters; + public SearchFinishedSubscriber(LocalSearchWaiters waiters) { this.waiters = waiters; } + @Override public void onMessage(Message message, byte[] pattern) { + waiters.signal(new String(message.getBody(), StandardCharsets.UTF_8)); + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboPolicySearchService.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboPolicySearchService.java new file mode 100644 index 0000000..7fc5cca --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboPolicySearchService.java @@ -0,0 +1,34 @@ +package com.arch.policy.search.infrastructure.rpc; + +import com.arch.policy.common.search.PolicySearchRequest; +import com.arch.policy.common.search.PolicySearchResponse; +import com.arch.policy.api.search.PolicySearchRpcService; +import com.arch.policy.search.application.AsyncSearchCoordinator; +import org.apache.dubbo.config.annotation.DubboService; +import org.springframework.beans.factory.annotation.Qualifier; + +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.Executor; + +@DubboService(version = "1.0.0", timeout = 3000) +public final class DubboPolicySearchService implements PolicySearchRpcService { + private final AsyncSearchCoordinator searchService; + private final Executor searchExecutor; + + public DubboPolicySearchService(AsyncSearchCoordinator searchService, + @Qualifier("policySearchExecutor") Executor searchExecutor) { + this.searchService = searchService; + this.searchExecutor = searchExecutor; + } + + @Override public CompletableFuture asyncSearch(final PolicySearchRequest request) { + final CompletableFuture future = new CompletableFuture(); + searchExecutor.execute(new Runnable() { + @Override public void run() { + try { future.complete(searchService.search(request)); } + catch (Exception failure) { future.completeExceptionally(failure); } + } + }); + return future; + } +} diff --git a/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboSupplierCallbackService.java b/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboSupplierCallbackService.java new file mode 100644 index 0000000..ae58dcb --- /dev/null +++ b/architecture/src/main/java/com/arch/policy/search/infrastructure/rpc/DubboSupplierCallbackService.java @@ -0,0 +1,23 @@ +package com.arch.policy.search.infrastructure.rpc; + +import com.arch.policy.search.application.SupplierCallbackService; +import com.arch.policy.common.search.CallbackResponse; +import com.arch.policy.common.search.SupplierCallbackRequest; +import com.arch.policy.api.search.SupplierCallbackRpcService; +import org.apache.dubbo.config.annotation.DubboService; + +@DubboService(version = "1.0.0", timeout = 3000) +public final class DubboSupplierCallbackService implements SupplierCallbackRpcService { + private final SupplierCallbackService callbackService; + public DubboSupplierCallbackService(SupplierCallbackService callbackService) { + this.callbackService = callbackService; + } + @Override public CallbackResponse callback(SupplierCallbackRequest request) { + try { + callbackService.callback(request); + return new CallbackResponse(true); + } catch (Exception failure) { + throw new IllegalStateException("supplier callback failed", failure); + } + } +} diff --git a/architecture/src/main/resources/application.yml b/architecture/src/main/resources/application.yml new file mode 100644 index 0000000..02c14bb --- /dev/null +++ b/architecture/src/main/resources/application.yml @@ -0,0 +1,41 @@ +spring: + application: + name: policy-search + kafka: + consumer: + enable-auto-commit: false + auto-offset-reset: earliest + key-deserializer: org.apache.kafka.common.serialization.StringDeserializer + value-deserializer: org.apache.kafka.common.serialization.StringDeserializer + listener: + ack-mode: record + redis: + host: localhost + port: 6379 + +dubbo: + application: + name: policy-search + protocol: + name: dubbo + port: 20880 + +book: + seata: + tenant-id: book + tx-service-group: book-saga-group + recovery: + scan-delay-ms: 5000 + creation: + scan-delay-ms: 3000 + +policy: + search: + redis-ttl-seconds: 60 + snapshot: + directory: ./data/policy-snapshots + initial-version: startup + retry-delay-ms: 5000 + kafka: + topic: policy-change + group-id: policy-search diff --git a/architecture/src/main/resources/db/book_order_schema.sql b/architecture/src/main/resources/db/book_order_schema.sql new file mode 100644 index 0000000..4de28ff --- /dev/null +++ b/architecture/src/main/resources/db/book_order_schema.sql @@ -0,0 +1,37 @@ +CREATE TABLE book_order_recovery_task ( + task_id BIGINT PRIMARY KEY AUTO_INCREMENT, + unique_key VARCHAR(256) NOT NULL, + order_no VARCHAR(64) NOT NULL, + child_order_no VARCHAR(64) NOT NULL DEFAULT '', + pnr VARCHAR(32) NOT NULL DEFAULT '', + task_type VARCHAR(32) NOT NULL, + task_status VARCHAR(32) NOT NULL, + attempts INT NOT NULL DEFAULT 0, + next_attempt_at TIMESTAMP NULL, + last_error VARCHAR(1024), + UNIQUE KEY uk_recovery_business (unique_key), + UNIQUE KEY uk_cancel_pnr (order_no, child_order_no, pnr, task_type) +); + +CREATE TABLE book_order_state_history ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + event_id VARCHAR(128) NOT NULL, + order_no VARCHAR(64) NOT NULL, + from_state VARCHAR(32) NOT NULL, + event_type VARCHAR(64) NOT NULL, + to_state VARCHAR(32) NOT NULL, + operator VARCHAR(64) NOT NULL, + occurred_at TIMESTAMP NOT NULL, + UNIQUE KEY uk_order_event (event_id) +); + +CREATE TABLE book_order_outbox ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + event_id VARCHAR(128) NOT NULL, + order_no VARCHAR(64) NOT NULL, + event_type VARCHAR(64) NOT NULL, + payload TEXT NOT NULL, + published TINYINT NOT NULL DEFAULT 0, + created_at TIMESTAMP NOT NULL, + UNIQUE KEY uk_outbox_event (event_id) +); diff --git a/architecture/src/main/resources/statelang/order_creation_saga.json b/architecture/src/main/resources/statelang/order_creation_saga.json new file mode 100644 index 0000000..4cf077f --- /dev/null +++ b/architecture/src/main/resources/statelang/order_creation_saga.json @@ -0,0 +1,68 @@ +{ + "Name": "TradeOrderCreationSaga", + "Comment": "本地子订单提交后扣库存、调用三方并按明确结果确认或返还库存", + "Version": "1.0.0", + "StartState": "DeductStock", + "States": { + "DeductStock": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "deductStock", + "Input": ["$.[order]", "$.[request]"], + "Output": {"stockDeducted": "$#root"}, + "Next": "RouteStockResult" + }, + "RouteStockResult": { + "Type": "Choice", + "Choices": [ + {"Expression": "$.[stockDeducted] == true", "Next": "CreateThirdOrder"} + ], + "Default": "CompleteStockFailure" + }, + "CreateThirdOrder": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "createThirdOrder", + "Input": ["$.[order]", "$.[request]"], + "Output": {"thirdResult": "$#root"}, + "Next": "RouteThirdResult" + }, + "RouteThirdResult": { + "Type": "Choice", + "Choices": [ + {"Expression": "$.[thirdResult].status.name() == 'SUCCESS'", "Next": "CompleteSuccess"}, + {"Expression": "$.[thirdResult].status.name() == 'FAILED'", "Next": "CompleteFailure"} + ], + "Default": "WaitForThirdResult" + }, + "CompleteSuccess": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "completeSuccess", + "Input": ["$.[order]"], + "Next": "Succeed" + }, + "CompleteFailure": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "completeFailure", + "Input": ["$.[order]"], + "Next": "Succeed" + }, + "CompleteStockFailure": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "completeStockFailure", + "Input": ["$.[order]"], + "Next": "Succeed" + }, + "WaitForThirdResult": { + "Type": "ServiceTask", + "ServiceName": "orderCreationSagaStateServices", + "ServiceMethod": "waitForThirdResult", + "Input": ["$.[order]"], + "Next": "Succeed" + }, + "Succeed": {"Type": "Succeed"} + } +} diff --git "a/note/Dubbo/Dubbo\345\272\225\345\261\202\346\272\220\347\240\201\345\255\246\344\271\240\357\274\210\344\272\224\357\274\211\342\200\224\342\200\224 Dubbo\347\232\204\346\263\250\345\206\214\344\270\255\345\277\203\351\207\215\350\257\225\346\234\272\345\210\266.md" "b/note/Dubbo/Dubbo\345\272\225\345\261\202\346\272\220\347\240\201\345\255\246\344\271\240\357\274\210\344\272\224\357\274\211\342\200\224\342\200\224 Dubbo\347\232\204\346\263\250\345\206\214\344\270\255\345\277\203\351\207\215\350\257\225\346\234\272\345\210\266.md" deleted file mode 100644 index e69de29..0000000 diff --git "a/note/Dubbo/Dubbo\345\272\225\345\261\202\346\272\220\347\240\201\345\255\246\344\271\240\357\274\210\345\233\233\357\274\211\342\200\224\342\200\224 Dubbo\347\232\204\346\263\250\345\206\214\344\270\255\345\277\203\347\274\223\345\255\230\346\234\272\345\210\266.md" "b/note/Dubbo/Dubbo\345\272\225\345\261\202\346\272\220\347\240\201\345\255\246\344\271\240\357\274\210\345\233\233\357\274\211\342\200\224\342\200\224 Dubbo\347\232\204\346\263\250\345\206\214\344\270\255\345\277\203\347\274\223\345\255\230\346\234\272\345\210\266.md" deleted file mode 100644 index e69de29..0000000 diff --git "a/note/Dubbo/Dubbo\346\272\220\347\240\201\346\220\255\345\273\272.md" "b/note/Dubbo/Dubbo\346\272\220\347\240\201\346\220\255\345\273\272.md" deleted file mode 100644 index 70b098d..0000000 --- "a/note/Dubbo/Dubbo\346\272\220\347\240\201\346\220\255\345\273\272.md" +++ /dev/null @@ -1,52 +0,0 @@ -## 前言 - -想要深入学习Dubbo,最好的方式就是阅读并调用Dubbo源码,接下来先来动手搭建一个Dubbo源码环境。 - -## 正文 - -### 1. 下载源码 - -步骤: - -1. 先从dubbo源码官网github中fork一份到自己的github仓库中。 - - ``` - git clone git@github.com:xxxxxxxx/dubbo.git - ``` - -2. 使用命令:git branch v2.7.8。 切换到分支2.7.8。 - - ``` - git checkout -b dubbo-2.7.8 dubbo-2.7.8 - ``` - -3. 导入方式(IDEA导入方式) - - 可以通过IDEA ——> File ——> Open ——> pom.xml ——> open as project - - 然后让IDEA下载相关的依赖,等下载完成即可。 - -4. mvn命令导入 - - ``` - mvn clean install -Dmaven.test.skip=true - ``` - - 然后执行下面的命令转换成 IDEA 项目: - - ``` - mvn idea:idea - ``` - - 如果执行报错了,则执行: - ``` - mvn idea:workspace - ``` - -### 2. 分支切换 - -本人Fork了官方Dubbo源码到本地仓库,并且新建了一个分支名为:SourceCode-of-Translation - -该分支主要用于进行源码注释,每个核心功能代码都有详细注释,欢迎大家Fork到本地,然后进行注释查看。 - -> 地址为:[SourceCode-of-Translation](https://github.com/coderbruis/dubbo) 下载到本地后,只需要切换到SourceCode-of-Translation分支即可。 diff --git "a/note/Dubbo/Dubbo\351\235\242\350\257\225\351\242\230.md" "b/note/Dubbo/Dubbo\351\235\242\350\257\225\351\242\230.md" deleted file mode 100644 index a8a3d5b..0000000 --- "a/note/Dubbo/Dubbo\351\235\242\350\257\225\351\242\230.md" +++ /dev/null @@ -1,22 +0,0 @@ -## 前言 - -Dubbo面试题 - -## 正文 - -RPC非常重要,很多人面试的时候都挂在了这个地方!你要是还不懂RPC是什么?他的基本原理是什么?你一定要把下边的内容记起来!好好研究一下!特别是文中给出的一张关于RPC的基本流程图,重点中的重点,Dubbo RPC的基本执行流程就是他,RPC框架的基本原理也是他,别说我没告诉你!看了下边的内容你要掌握的内容如下,当然还有很多: - -1. RPC的由来,是怎么一步一步演进出来的; -2. RPC的基本架构是什么; -3. RPC的基本实现原理, 重点; -4. REST和SOAP、PRC的区别; -5. 整个调用的过程经历了哪几部?和SpringMVC流程区别? - - -### 1. 为什么要有RPC - -随着互联网的发展,网站应用的规模不断扩大,常规的垂直应用架构已无法应对,分布式服务架构以及流动计算架构势在必行,亟需一个治理系统确保架构有条不紊的演进。 - -- [PRC原理图](https://github.com/coderbruis/JavaSourceLearning/blob/master/images/PRC/rpc.jpg) - - diff --git "a/note/Dubbo/dubbo\345\272\225\345\261\202\345\216\237\347\220\206\351\230\205\350\257\273\351\241\272\345\272\217.md" "b/note/Dubbo/dubbo\345\272\225\345\261\202\345\216\237\347\220\206\351\230\205\350\257\273\351\241\272\345\272\217.md" deleted file mode 100644 index 18929b7..0000000 --- "a/note/Dubbo/dubbo\345\272\225\345\261\202\345\216\237\347\220\206\351\230\205\350\257\273\351\241\272\345\272\217.md" +++ /dev/null @@ -1,11 +0,0 @@ -## 前言 - -在真正深入Dubbo底层源码前,先梳理好阅读路线,这样才能够事半功倍。 - -## 正文 - -### 1. 配置加载流程? - -[配置加载流程](http://dubbo.apache.org/zh-cn/docs/user/configuration/configuration-load-process.html) - -### 2. \ No newline at end of file diff --git "a/note/JDK/AQS\346\272\220\347\240\201\345\210\206\346\236\220.md" "b/note/JDK/AQS\346\272\220\347\240\201\345\210\206\346\236\220.md" new file mode 100644 index 0000000..8fb00bc --- /dev/null +++ "b/note/JDK/AQS\346\272\220\347\240\201\345\210\206\346\236\220.md" @@ -0,0 +1,1060 @@ +AQS版本:JDK 1.8 + +# AQS灵魂三问 +## 是什么? +AQS 全称是 AbstractQueuedSynchronizer,中文通常翻译为:**抽象队列同步器**。 + +AQS 本身不是一把可以直接使用的锁,而是一个用于构建锁和同步器的基础框架。 + +JDK 中很多并发工具都基于 AQS 实现: + +```java +ReentrantLock +ReentrantReadWriteLock +Semaphore +CountDownLatch +ThreadPoolExecutor.Worker +``` + +## 解决什么问题? +AQS 主要解决两类通用问题: + +1. 如何通过一个同步状态 state 判断资源是否可获取 +2. 获取失败后,如何让线程排队、阻塞,并在合适时机被唤醒 + + + +AQS 用于统一解决线程竞争、等待队列、阻塞与唤醒问题。 + +## 起什么作用? +AQS是 ReentrantLock、Semaphore 等同步器的基础框架。 + +AQS 将同步器拆成了两部分: + +1. 子类负责: + 1. 定义 state 的业务含义。 + 2. 定义获取资源和释放资源的规则。 +2. AQS负责: + 1. CAS 修改状态。 + 2. 维护等待队列。 + 3. 阻塞线程。 + 4. 唤醒线程。 + 5. 处理中断、超时和取消。 + +例如: + +1. ReentrantLock:state 表示锁的重入次数 +2. CountDownLatch:state 表示剩余计数 +3. Semaphore:state 表示剩余许可证数量 + +**** + +**AQS 的核心内容可以浓缩为:** + +1. 一个 volatile int state +2. 一个双向同步等待队列 +3. CAS 原子操作 +4. LockSupport.park/unpark 阻塞与唤醒 + +# AQS整体架构 +AQS 支持两种资源获取模式: + +1. 独占模式 Exclusive:同一时刻只允许一个线程成功获取资源。 +2. 共享模式 Shared:同一时刻允许多个线程成功获取资源。 + + + +典型应用: + +1. 独占模式:ReentrantLock、ReentrantReadWriteLock 写锁。 +2. 共享模式:Semaphore、CountDownLatch、ReentrantReadWriteLock 读锁。 + + + +两种模式共用同一个同步队列: + +```plain +head tail + | | + v v +[哨兵节点] <-> [独占节点 T1] <-> [共享节点 T2] <-> [独占节点 T3] +``` + + + +AQS 不理解“锁”“许可证”“计数器”等业务含义。它只负责: + +1. tryAcquire/tryAcquireShared 成功:线程继续执行。 +2. tryAcquire/tryAcquireShared 失败:线程进入队列并阻塞。 +3. tryRelease/tryReleaseShared 成功:唤醒后继线程。 + + + + +![](https://cdn.nlark.com/yuque/0/2026/png/12890164/1783689319993-5d064d1b-9d0e-4f6f-b208-5fbaedbc0952.png) + + + +# AQS核心结论 +AQS 本质上是一个用于构建锁和同步器的基础框架,核心由两部分组成:一个表示同步状态的 state,以及一个保存等待线程的 FIFO 双向队列。 + + + +线程获取资源时,首先通过 CAS 尝试修改 state: + ++ 获取成功:线程继续执行。 ++ 获取失败:线程封装成 Node 加入等待队列,并通过 LockSupport.park() 阻塞。 ++ 资源释放:修改 state,再通过 LockSupport.unpark() 唤醒后继节点重新竞争。 + + + +AQS 自身不定义资源如何获取和释放,而是通过模板方法交给子类实现: + ++ tryAcquire() ++ tryRelease() ++ tryAcquireShared() ++ tryReleaseShared() + + + +因此,AQS 的核心可以概括为: + ++ state 表示资源或锁状态。 ++ CAS 保证状态修改的原子性。 ++ 等待队列管理获取资源失败的线程。 ++ park/unpark 实现线程阻塞与唤醒。 ++ 模板方法定义具体同步规则。 ++ 支持独占模式和共享模式。 ++ ConditionObject 提供条件等待队列。 + + + +AQS 将“资源竞争、线程排队、阻塞与唤醒”统一封装,子类只需要实现资源的获取和释放规则。 + +# AQS 的重要成员变量和内部类 +## state:同步状态 +state 是 AQS 中最核心的变量: + +```java +private volatile int state; +``` + +AQS 使用一个 volatile int 保存同步状态,并通过以下方法读取和修改: + +```java +protected final int getState() +protected final void setState(int newState) +protected final boolean compareAndSetState(int expect, int update) +``` + +其中: + +```plain +getState():读取当前同步状态 +setState():直接设置同步状态 +compareAndSetState():通过 CAS 原子更新同步状态 +``` + +state 的含义由具体同步器决定。 + + + +ReentrantLock 中的 state表示 + ++ state = 0:锁未被任何线程持有 ++ state = 1:当前线程第一次获取锁 ++ state = 2:当前线程重入一次 ++ state = 3:当前线程重入两次 + + + +Semaphore 中的 state表示当前剩余许可证数量。 + +例如: + +```plain +初始许可证数量为 3 + +state = 3:还可以有 3 个线程获取 +state = 2:还可以有 2 个线程获取 +state = 0:没有剩余许可证 +``` + + + +CountDownLatch中的state表示尚未完成的计数。 + +例如: + +```plain +new CountDownLatch(3) + +初始 state = 3 +每次 countDown(),state 减 1 +state = 0 时,所有 await() 线程可以继续执行 +``` + +**** + +**总结:AQS 只提供一个线程安全的状态字段,不规定状态的业务含义。** + + + +## head 和 tail:同步队列首尾节点 +```java +private transient volatile Node head; +private transient volatile Node tail; +``` + +AQS 使用一个基于 CLH 思想改造的双向链表维护获取资源失败的线程。 + +```plain +head tail + | | + v v +[哨兵节点] <-> [节点 T1] <-> [节点 T2] <-> [节点 T3] +``` + +其中: + +```plain +head:队列头节点,通常是已经获取过资源的哨兵节点 + +tail:队列尾节点,新节点通过 CAS 追加到 tail +``` + +head和tail都使用 volatile 修饰,保证不同线程之间的可见性。 + + + +AQS 创建时不会立刻创建队列。只有第一次发生竞争、线程需要入队时,才会初始化一个空的哨兵节点: + +```plain +head == tail == new Node() +``` + +这样可以避免在从未发生竞争的同步器上浪费节点对象。 + +## Node:同步队列节点 +AQS 内部通过Node表示一个等待线程。 + +源码结构如下: + +```java +static final class Node { + ... + static final Node SHARED = new Node(); + static final Node EXCLUSIVE = null; + + static final int CANCELLED = 1; + static final int SIGNAL = -1; + static final int CONDITION = -2; + static final int PROPAGATE = -3; + + volatile int waitStatus; + // 指向同步队列中的前驱节点。 + volatile Node prev; + volatile Node next; + // 保存当前节点对应的等待线程。 + // 当节点需要被唤醒时,AQS 会执行:LockSupport.unpark() + volatile Thread thread; + Node nextWaiter; + ... +} +``` + +### prev +prev表示指向同步队列中的前驱节点。 + +```java +volatile Node prev; +``` + +指向同步队列中的前驱节点。 + +AQS 判断当前节点能否尝试获取资源时,最重要的条件是:**当前节点的前驱节点是否是 head。** + +只有队列中的第一个有效等待节点,才有资格再次调用tryAcquire()或tryAcquireShared()竞争资源。 + + + +AQS中判断源码如下: + +独占模式acquire: + +```java +final boolean acquireQueued(final Node node, int arg) { + ... + final Node p = node.predecessor(); + if (p == head && tryAcquire(arg)) { + ... + } + ... +} +``` + +共享模式acquire: + +```java +private void doAcquireShared(int arg) { + ... + final Node p = node.predecessor(); + if (p == head) { + ... + } + ... +} +``` + + + +**所以“当前节点的前驱节点是否是 head”是非常重要的一个条件。** + + + +### next(设计思想非常的细节) +指向同步队列中的后继节点。 + +```java +volatile Node next; +``` + +释放资源时,AQS 通常通过 head.next 找到需要唤醒的线程。 + +但next只是一个优化路径,不是绝对可靠的队列判断依据。 + +原因是节点入队过程如下: + +1. node.prev = oldTail +2. CAS 把 tail 从 oldTail 修改为 node +3. oldTail.next = node + +```java +// 入队操作,并返回当前node的前序节点 +private Node enq(final Node node) { + for (;;) { + Node t = tail; + if (t == null) { + if (compareAndSetHead(new Node())) + tail = head; + } else { + // 1. oldTail <- node + node.prev = t; + // 2. oldTail <- node(tail) 将node赋值给tail,表明当前node成为了tail + if (compareAndSetTail(t, node)) { + // oldTail -> node(tail) + t.next = node; + // 返回oldTail + return t; + } + } + } +} +``` + + + +在线程完成第 2 步、尚未完成第 3 步时: + +```plain +tail 已经指向新节点 +但 oldTail.next 仍然是 null +``` + +因此在部分场景下,如果判断到next=null,则AQS 会从tail沿着prev反向扫描,寻找有效后继节点。 + +```java +private void unparkSuccessor(Node node) { + ... + Node s = node.next; + if (s == null || s.waitStatus > 0) { + s = null; + // 通过next找不到要unpark的节点,则从tail开始向前遍历 + for (Node t = tail; t != null && t != node; t = t.prev) + if (t.waitStatus <= 0) + s = t; + } + ... +} +``` + + + +**总结:**因为有 next 指针,所以 release 时可以直接取 head,再通过 head.next 找到要唤醒的后继节点,通常是 O(1)。而如果没有next指针,则需要从tail尾结点向前通过prev遍历,时间复杂度为O(n)。所以在入队维护next指针的时候,因为: + +1)node.prev = t; + +2)compareAndSetTail(t, node) + +3)t.next = node; + +导致可能t.next会存在null的场景,所以通过next指针遍历会拿不到数据。所以如果next为null,就会退化成通过prev指针去获取目标节点。 + + + +### nextWaiter +nextWaiter 有两个用途。 + +#### 用途一:标识节点模式 +```java +nextWaiter == Node.SHARED:共享节点 +nextWaiter == Node.EXCLUSIVE:独占节点 +``` + +其中 Node.EXCLUSIVE 实际是 null。 + +#### 用途二:连接 Condition 条件队列 +Condition 条件队列是单向链表,节点之间通过 `nextWaiter` 连接: + +```plain +firstWaiter + | + v +[T1 CONDITION] -> [T2 CONDITION] -> [T3 CONDITION] + ^ + | + lastWaiter +``` + +所以: + +```plain +prev/next:用于 AQS 同步队列 +nextWaiter:用于 Condition 条件队列,或者标记共享模式 +``` + + + +### waitStatus 状态详解 +Node 中最难理解的字段是: + +```java +volatile int waitStatus; +``` + +它表示节点当前的等待状态。 + +JDK 8 中主要有以下几种状态: + ++ CANCELLED = 1 ++ SIGNAL = -1 ++ CONDITION = -2 ++ PROPAGATE = -3 ++ 默认状态 = 0 + +**** + +#### 0:默认状态 +新创建的同步队列节点,`waitStatus` 默认为 0。 + +表示节点当前没有特殊状态。 + + + +#### CANCELLED = 1:节点已取消 +```java +static final int CANCELLED = 1; +``` + +出现以下情况时,节点可能被取消: + ++ 线程等待超时 ++ 线程在可中断等待中被中断 ++ 获取资源过程中发生异常 + + + +取消后的节点不会再次参与资源竞争。 + +AQS 会在后续遍历中跳过 waitStatus > 0 的节点。 + +需要注意:CANCELLED 是唯一的正数状态。 + +因此源码中经常通过下面的方式判断节点是否已取消:waitStatus > 0 + + + +#### SIGNAL = -1:后继节点需要被唤醒 +```java +static final int SIGNAL = -1; +``` + +**这是同步队列中最重要的状态。** + + + +假设队列结构如下:pred -> node + +当 pred.waitStatus == SIGNAL 时,表示: + +```plain +node 准备阻塞。 +当 pred 释放资源、成为无效头节点或被取消时,pred 负责唤醒 node。 +``` + +一个非常容易混淆的点是: + +**SIGNAL 状态保存在前驱节点上,但它表达的是后继节点需要被唤醒。** + +例如: + +```plain +head(waitStatus = SIGNAL) -> T1 +``` + +表示 T1 可以安全阻塞,head 对应的资源持有者释放资源时,需要唤醒 T1。 + + + +为什么不把 SIGNAL 放在当前节点上? + +因为释放资源时,释放线程主要操作当前 `head`,由前驱节点记录唤醒责任,可以减少对后继节点状态的竞争修改。 + + + +#### CONDITION = -2:节点正在 Condition 条件队列中等待 +```java +static final int CONDITION = -2; +``` + +线程调用 Condition.await() 后,会进入 Condition 条件队列,此时节点状态为 CONDITION。 + +```plain +Condition 条件队列: + +[T1 CONDITION] -> [T2 CONDITION] -> [T3 CONDITION] +``` + + + +当其他线程调用 signal() 后,节点会: + +```plain +waitStatus:CONDITION -> 0 +从 Condition 条件队列转移到 AQS 同步队列 +``` + +只有重新进入同步队列并再次获取锁后,`await()` 才会返回。 + + + +#### PROPAGATE = -3:共享模式继续传播 +```java +static final int PROPAGATE = -3; +``` + +PROPAGATE 只用于共享模式。 + +它用于记录:即使当前释放动作没有直接找到需要唤醒的节点,后续共享节点仍需要继续检查并传播唤醒。 + + + +该状态主要用于解决共享模式下并发获取、释放交错时可能出现的传播遗漏问题。 + +可以先把它理解为:共享唤醒传播标记。 + + + +## waitStatus 的正负设计 +AQS 中对 `waitStatus` 的判断非常精简: + +```plain +waitStatus > 0:节点已取消,需要跳过 +waitStatus < 0:节点处于有效的信号、条件或传播状态 +waitStatus = 0:普通初始状态 +``` + +这种设计让很多分支只需要判断正负,而不必逐个比较状态常量。 + + + +# AQS 同步队列为什么是 CLH 变体 +AQS 的同步队列通常被称为 CLH 队列,但它并不是原始的 CLH 自旋锁队列。 + +原始 CLH 队列的核心思想是:每个线程关注自己的前驱节点状态。 + + + +AQS 保留了这个思想,但做了改造: + +```plain +原始 CLH:线程主要自旋等待 +AQS:线程尝试几次后,通过 LockSupport.park 阻塞 + +原始 CLH:通常只需要前驱引用 +AQS:增加 prev 和 next,便于取消、唤醒和队列维护 +``` + + + +AQS 节点主要观察前驱节点: + +```plain +前驱是 head:尝试获取资源 +前驱为 SIGNAL:当前线程可以安全 park +前驱已取消:跳过取消节点,重新连接有效前驱 +``` + +因此,AQS 同步队列可以理解为:基于 CLH 思想改造的 FIFO 双向阻塞队列 + + + +# AQS 留给子类实现的五个方法 +AQS 将排队和阻塞机制实现好,但资源获取规则需要子类定义。 + +核心扩展方法如下: + +```java +protected boolean tryAcquire(int arg) +protected boolean tryRelease(int arg) +protected int tryAcquireShared(int arg) +protected boolean tryReleaseShared(int arg) +protected boolean isHeldExclusively() +``` + +默认实现都会抛出UnsupportedOperationException,子类根据需要选择实现独占模式、共享模式或两者。 + + + +# AQS多线程竞争入队流程 +## 独占锁场景 +下面按**独占锁**场景流程,比如ReentrantLock.lock()。 + +**背景:当前有T0、T1、T2三个线程竞争锁,默认T0已获得锁。** + +### 1. T0 已经持有锁 +```plain +state = 1 +owner = T0 + +同步队列:空 +head = null +tail = null +``` + +**** + +### 2. T1 来竞争,tryAcquire 失败,封装成 Node 入队 +AQS 执行:addWaiter(Node.EXCLUSIVE) + + + +队列第一次初始化,会先创建哨兵 head: + +```plain +head(dummy) <-> T1 + ^ + tail +``` + +此时节点状态: + +```plain +head.waitStatus = 0 +T1.waitStatus = 0 +``` + +### 3. T1 入队后马上自旋再次竞争,失败,准备阻塞 +T1 在 acquireQueued 里判断: + +```plain +p == head && tryAcquire(arg) +``` + +此时 `p == head`,但锁还被 T0 持有,所以失败。 + +然后进入: + +```plain +shouldParkAfterFailedAcquire(head, T1) +``` + +T1 会把自己的前驱 head 设置成 **SIGNAL状态**: + +```plain +head(SIGNAL) <-> T1(0) + ^ + tail +``` + + + +然后T1发现前序节点是SIGNAL状态,然后T1调用park()进入阻塞状态。 + + + +### 4. T2 也来竞争,失败后入队 +```plain +head(SIGNAL) <-> T1(0) <-> T2(0) + ^ + tail +``` + +T2 的前驱是 T1。T2 获取失败后,马上自旋再次竞争锁,失败之后会把前驱 T1 设置成 `SIGNAL`: + +```plain +head(SIGNAL) <-> T1(SIGNAL) <-> T2(0) + ^ + tail +``` + +然后T2发现前序节点是SIGNAL状态,然后T2调用park()进入阻塞状态。 + +### 5. T0 释放锁,AQS 从 head 唤醒后继 +T0 调用: + +```plain +unlock() + -> release(1) + -> tryRelease(1) + -> unparkSuccessor(head) +``` + +队列: + +```plain +head(SIGNAL) <-> T1(SIGNAL) <-> T2(0) +``` + +AQS 找: + +```plain +head.next == T1 +``` + +然后: + +```plain +LockSupport.unpark(T1.thread) +``` + +结果: + +```plain +T1 被唤醒 +T2 仍阻塞 +``` + +### 6. T1 被唤醒后重新竞争锁,成功 +T1 醒来后回到循环: + +```plain +p == head && tryAcquire(arg) +``` + +此时锁已经空了,所以 T1 成功获取锁。 + +然后: + +```plain +setHead(T1) +``` + +队列变成: + +```plain +head = T1 +T1(thread=null, prev=null, SIGNAL) <-> T2(0) + ^ + tail +``` + +旧的 dummy head 会断开,帮助 GC: + +```plain +T1 成为新的 head +``` + +注意: + +```plain +T1 成为 head 后,它本身不再代表一个等待线程; +它代表当前成功获取过锁的节点。 +``` + +### 7. T1 释放锁,再唤醒 T2 +T1 执行: + +```plain +unlock() + -> release(1) + -> unparkSuccessor(head) +``` + +此时: + +```plain +head(T1, SIGNAL) <-> T2(0) +``` + +AQS 唤醒: + +```plain +head.next == T2 +``` + +T2 醒来,重新执行: + +```plain +p == head && tryAcquire(arg) +``` + +成功后: + +```plain +setHead(T2) +``` + +队列变成: + +```plain +head = T2 +tail = T2 +``` + +如果后面没有等待节点: + +```plain +T2.waitStatus 通常是 0 +``` + +# AQS核心方法 +## acquire() +当某个线程发起锁获取,比如调用ReentrantLock.lock()方法时,调用链会走到AbstractQueuedSynchronizer.acquire()方法,源码如下: + +```java +public final void acquire(int arg) { +if (!tryAcquire(arg) && + acquireQueued(addWaiter(Node.EXCLUSIVE), arg)) + selfInterrupt(); +} +``` + +## nonfairTryAcquire() +tryAcquire()逻辑在AQS子类里,当前就是在ReentrantLock中,核心逻辑: + +```java +final boolean nonfairTryAcquire(int acquires) { + final Thread current = Thread.currentThread(); + // 获取AQS状态 + int c = getState(); + // 没有线程持有这个共享状态 + if (c == 0) { + // CAS 变更state状态 + if (compareAndSetState(0, acquires)) { + // 标识当前线程持有state + setExclusiveOwnerThread(current); + return true; + } + } + // 如果是持有锁的线程 + else if (current == getExclusiveOwnerThread()) { + // state加一,重复持有锁 + int nextc = c + acquires; + if (nextc < 0) // overflow + throw new Error("Maximum lock count exceeded"); + setState(nextc); + return true; + } + return false; +} +``` + +nonfairTryAcquire()方法比较简单,主要就是通过CAS变更state状态,然后将持有锁的线程标识为owner,后续竞争锁则判断该线程是否持有,否则加锁失败返回false。 + + + +## addWaiter() +```java + private Node addWaiter(Node mode) { + Node node = new Node(Thread.currentThread(), mode); + // 尝试一次快速入队 + Node pred = tail; + // 如果pred=null,则表示队列还未初始化,快速入队逻辑不负责初始化 + if (pred != null) { + node.prev = pred; + // 并发竞争失败,会走到兜底入队逻辑 + if (compareAndSetTail(pred, node)) { + pred.next = node; + return node; + } + } + // 兜底入队逻辑 + enq(node); + return node; + } +``` + +addWaiter() 先用一次 CAS 快速入队;如果队列未初始化或发生并发竞争,就交给 enq() 自旋,确保当前线程节点最终进入 AQS 同步队列。 + + + +enq() 方法负责同步队列的延迟初始化,以及节点竞争入队。当队列未初始化时,通过 CAS 创建哨兵头节点;当队列已经初始化时,通过 CAS 将当前节点设置为新的尾节点。如果 CAS 失败,则不断自旋重试,直到节点成功入队。 + +```java + private Node enq(final Node node) { + for (;;) { + Node t = tail; + // 队列初始化 + if (t == null) { // Must initialize + if (compareAndSetHead(new Node())) + tail = head; + } else { + node.prev = t; + // CAS往队列尾部添加节点 + if (compareAndSetTail(t, node)) { + t.next = node; + // !返回的是入队节点的前序节点 + return t; + } + } + } + } +``` + +## acquireQueued() + + +```java +/** + * 已入队节点以独占模式获取同步状态。 + * + * @return 等待过程中是否发生过中断 + */ +final boolean acquireQueued(final Node node, int arg) { + boolean failed = true; + try { + boolean interrupted = false; + + // 自旋,直到成功获取同步状态 + for (;;) { + final Node p = node.predecessor(); + + // 只有头节点的直接后继才有资格尝试获取同步状态 + if (p == head && tryAcquire(arg)) { + // 获取成功,当前节点成为新的头节点 + setHead(node); + p.next = null; // 断开旧头节点,帮助 GC + failed = false; + return interrupted; + } + + // 获取失败,判断是否需要阻塞;被唤醒后检查中断状态 + if (shouldParkAfterFailedAcquire(p, node) && + parkAndCheckInterrupt()) { + interrupted = true; + } + } + } finally { + // 出现异常等获取失败的情况时,取消当前节点 + if (failed) { + cancelAcquire(node); + } + } +} +``` + +shouldParkAfterFailedAcquire()方法核心就是将入参node的前驱节点状态置为:SIGNAL,然后方法返回false,继续执行后面的parkAndCheckInterrupt(),将node节点置为阻塞状态。 + + + +这里就是对应着有新线程竞争锁失败之后,先加入队列,之后自旋尝试再次竞争锁,失败了则将前序节点置为SIGNAL,然后将自己阻塞等待被唤醒。 + + + +下面再分析下释放锁的流程,释放锁核心会调用AQS的release()方法。 + +```java +public final boolean release(int arg) { + // 子类负责更新锁状态,并判断锁是否已完全释放 + if (tryRelease(arg)) { + Node h = head; + + // 头节点存在且后继节点需要唤醒 + if (h != null && h.waitStatus != 0) + unparkSuccessor(h); + + return true; + } + + // 重入次数尚未归零,本次只减少持锁次数 + return false; +} +``` + + + +tryRelease()方法在ReentrantLock中,源码如下: + +```java +protected final boolean tryRelease(int releases) { + // 减少重入计数 + int c = getState() - releases; + + // 只有锁的持有线程才能释放锁 + if (Thread.currentThread() != getExclusiveOwnerThread()) + throw new IllegalMonitorStateException(); + + boolean free = false; + + // 重入计数归零,锁才算完全释放 + if (c == 0) { + free = true; + setExclusiveOwnerThread(null); + } + + // 更新剩余重入次数 + setState(c); + return free; +} +``` + + + +AQS的unparkSuccessor()源码如下: + +```java +/** + * 唤醒等待队列中有效的后继节点。 + */ +private void unparkSuccessor(Node node) { + int ws = node.waitStatus; + + // 清除当前节点的待唤醒标记 + if (ws < 0) + compareAndSetWaitStatus(node, ws, 0); + + Node s = node.next; + + // next 无效时,从队尾反向查找最靠前的有效等待节点 + if (s == null || s.waitStatus > 0) { + s = null; + for (Node t = tail; t != null && t != node; t = t.prev) { + // waitStatus > 0 表示节点已经取消 + if (t.waitStatus <= 0) + s = t; + } + } + + // 唤醒目标线程,让其重新参与锁竞争 + if (s != null) + LockSupport.unpark(s.thread); +} +``` + + + +释放锁流程总结: + +1. 当前线程调用 unlock(),最终进入 AQS 的 release(1)。 +2. release() 调用 tryRelease(1),减少 state 表示的重入次数。 +3. 如果当前线程不是锁的持有者,抛出 IllegalMonitorStateException。 +4. 如果 state 仍大于 0,说明当前线程还持有重入锁,不唤醒其他线程。 +5. 如果 state 减少到 0,清空锁的持有线程,表示锁已完全释放。 +6. release() 检查等待队列,通过 unparkSuccessor() 找到有效的后继节点。 +7. 调用 LockSupport.unpark() 唤醒对应线程,使其重新尝试获取锁。 + + + +需要注意:unpark() 只是让等待线程具备继续运行的条件,并不代表它立刻获得锁。线程被唤醒后,仍然需要参与锁竞争。 + diff --git "a/note/JDK/HashMap\346\272\220\347\240\201\345\210\206\346\236\220.md" "b/note/JDK/HashMap\346\272\220\347\240\201\345\210\206\346\236\220.md" new file mode 100644 index 0000000..a586cb2 --- /dev/null +++ "b/note/JDK/HashMap\346\272\220\347\240\201\345\210\206\346\236\220.md" @@ -0,0 +1,634 @@ ++ 当前HashMap版本:JDK1.8 ++ 转载请标明出处 + +# HashMap底层原理图 +![HashMap原理图](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/HashMap.png) + +# HashMap的重要成员变量以及内部类 +> **默认容量:DEFAULT_INITIAL_CAPACITY** +> + +```java +static final int DEFAULT_INITIAL_CAPACITY = 1 << 4; // aka 16 +``` + +当HashMap没有设置大小时,调用HashMap的put方法时,会进行初始值,并使用DEFAULT_INITIAL_CAPACITY设置默认大小。调用位置在: + +```java +final Node[] resize() { + ... + else { // zero initial threshold signifies using defaults + // HashMap默认大小,16 + newCap = DEFAULT_INITIAL_CAPACITY; + // 扩容阈值,12 + newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY); + } + ... +} +``` + + + +> **扩容阈值:DEFAULT_LOAD_FACTOR** +> + +```java +static final float DEFAULT_LOAD_FACTOR = 0.75f; +``` + +在resize()中可以看到,当HashMap首次添加元素,调用put时,会计算第一次扩容阈值12,也就是说HashMap中元素=12即触发扩容。 + + + +扩容实际发生在putVal()中,源码如下: + +```java +final V putVal(int hash, K key, V value, boolean onlyIfAbsent, + boolean evict) { + ... + if (++size > threshold) + resize(); + ... +} +``` + +当HashMap中添加了新元素,size递增之后判断是否大于扩容阈值。 + + + +> **树化相关配置:TREEIFY_THRESHOLD、UNTREEIFY_THRESHOLD、MIN_TREEIFY_CAPACITY** +> + +```java +// 桶内链表长度达到 8,考虑红黑树化 +static final int TREEIFY_THRESHOLD = 8; +// 数组容量至少 64,才真正允许红黑树化 +static final int UNTREEIFY_THRESHOLD = 6; +// 红黑树节点减少到 6 个或更少,考虑退化回链表 +static final int MIN_TREEIFY_CAPACITY = 64; +``` + +链表长到 8 时,如果数组容量小于 64,先扩容;如果数组容量已经至少 64,才转红黑树。树节点少到 6 时,再退化回链表。 + + + +```java +final V putVal(int hash, K key, V value, boolean onlyIfAbsent, + boolean evict) { + ... + if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st + treeifyBin(tab, hash); + ... +} + +final void treeifyBin(Node[] tab, int hash) { + ... + if (tab == null || (n = tab.length) < MIN_TREEIFY_CAPACITY) + // 先扩容 + resize(); + else if ((e = tab[index = (n - 1) & hash]) != null) { + // 红黑树化 + } + ... +} +``` + +可以看到,当HashMap中元素小于MIN_TREEIFY_CAPACITY时,是先进行的扩容,而非直接红黑树化。 + + + +> **HashMap内部类Node** +> + +这个类就是HashMap桶中存储的基本单元类,HashMap其实可以理解为一个数组,数组里每个位置叫一个桶bucket。 + +```java +static class Node implements Map.Entry { + final int hash; + final K key; + V value; + Node next; +} +``` + +在HashMap中通过一个Node数组来存储Node节点。 + +```java +transient Node[] table; +``` + +如果多个 key 经过 hash 计算后落到同一个桶,就会通过 `next` 串起来,形成链表: + +```plain +table[3] + | + v +Node(key1, value1) + | + next + v +Node(key2, value2) + | + next + v +Node(key3, value3) +``` + +HashMap 的每个桶位保存一个头节点引用;发生 hash 冲突时,新节点通过 next 挂在这个桶的链表后面;链表过长时可能转成红黑树。 + +还有一个细节需要注意,Node不仅存了key、value最核心的键值对信息,还存储了这个Node的hash值,这个hash值的作用是:**用于快速比较、查找、扩容迁移和树化查找,避免反复计算 hashCode,也保证节点定位稳定。** + +hash核心作用代码: + +```java +final V putVal(int hash, K key, V value, boolean onlyIfAbsent, + boolean evict) { + ... + Node e; K k; + if (p.hash == hash && + ((k = p.key) == key || (key != null && key.equals(k)))) + e = p; + ... +} +``` + +putVal() 里比较 key 是否已经存在,这里先比较hash值,如果hash值不一样,则key一定不相等,可以直接跳过,避免频繁调用 equals()。 + + + +查找时也会用到hash: + +```java +final Node getNode(int hash, Object key) { + ... + if (first.hash == hash && // always check first node + ((k = first.key) == key || (key != null && key.equals(k)))) + return first; + ... +} +``` + +**总结:Node.hash 是 key 的缓存 hash 值,用于快速比较、查找、扩容迁移和树化查找,避免反复计算 hashCode,也保证节点定位稳定。** + + + + + + + +# HashMap核心方法源码分析 +## putVal() +```java +final V putVal(int hash, K key, V value, boolean onlyIfAbsent, + boolean evict) { + + // tab:HashMap 底层数组 + // p:当前桶的第一个节点 + // n:数组长度 + // i:key 对应的桶下标 + Node[] tab; Node p; int n, i; + + // 如果 table 还没初始化,或者长度为 0,则先 resize 初始化数组 + // new HashMap<>() 第一次 put 时,会在这里创建默认长度 16 的数组 + if ((tab = table) == null || (n = tab.length) == 0) + n = (tab = resize()).length; + + // 计算桶下标:(n - 1) & hash + // 因为 n 是 2 的幂,所以等价于 hash % n,但效率更高 + // 如果这个桶为空,直接放入新 Node + if ((p = tab[i = (n - 1) & hash]) == null) + tab[i] = newNode(hash, key, value, null); + + else { + // e:最终找到的旧节点;如果为 null,说明是新增 key + // k:临时保存已有节点的 key + Node e; K k; + + // 先检查桶中第一个节点是否就是目标 key + // 先比 hash,再比 key 引用或 equals + if (p.hash == hash && + ((k = p.key) == key || (key != null && key.equals(k)))) + e = p; + + // 如果桶已经是红黑树结构,走红黑树插入/查找逻辑 + else if (p instanceof TreeNode) + e = ((TreeNode)p).putTreeVal(this, tab, hash, key, value); + + else { + // 桶是普通链表,遍历链表 + for (int binCount = 0; ; ++binCount) { + + // 如果下一个节点为空,说明没有找到相同 key + // 把新节点追加到链表尾部 + if ((e = p.next) == null) { + p.next = newNode(hash, key, value, null); + + // 链表长度达到树化阈值 8 时,尝试树化 + // 注意:treeifyBin 内部还会判断 table 长度是否 >= 64 + // 如果小于 64,优先扩容,不会真正树化 + if (binCount >= TREEIFY_THRESHOLD - 1) + treeifyBin(tab, hash); + + break; + } + + // 找到 hash 和 key 都相同的旧节点,停止遍历 + if (e.hash == hash && + ((k = e.key) == key || (key != null && key.equals(k)))) + break; + + // 继续向后遍历链表 + p = e; + } + } + + // e != null 表示找到了旧 key,不是新增,而是更新 value + if (e != null) { + V oldValue = e.value; + + // onlyIfAbsent 为 false:直接覆盖旧值 + // onlyIfAbsent 为 true:只有旧值为 null 时才覆盖 + // put() 传 false,putIfAbsent() 传 true + if (!onlyIfAbsent || oldValue == null) + e.value = value; + + // LinkedHashMap 扩展点:访问节点后回调 + // HashMap 中是空实现 + afterNodeAccess(e); + + // 返回旧值 + return oldValue; + } + } + + // 结构性修改次数 +1 + // 用于 fail-fast,比如迭代时检测并发修改 + ++modCount; + + // 新增节点后 size +1 + // 如果 size 超过扩容阈值 threshold,则扩容 + if (++size > threshold) + resize(); + + // LinkedHashMap 扩展点:插入节点后回调 + // HashMap 中是空实现 + afterNodeInsertion(evict); + + // 新增 key 时返回 null + return null; +} +``` + +从源码中可以看到几个细节。 + +### 1)**JDK8 的 HashMap 链表插入用的是尾插法**。 +```java +if ((e = p.next) == null) { + p.next = newNode(hash, key, value, null); + ... + break; +} +``` + +```plain +table[i] -> A -> B +``` + +插入新元素之后 + +```plain +table[i] -> A -> B -> C +``` + +对比 JDK7,JDK7 HashMap 扩容迁移时使用头插法,可能在并发扩容下形成环链表,导致死循环。JDK8 改了扩容迁移逻辑,并且普通链表插入也是尾插,能保持链表相对顺序。 + +JDK8 HashMap 仍然不是线程安全的。尾插法解决不了所有并发问题。并发 put 仍可能出现数据覆盖、丢数据、size 不准、扩容状态异常等问题。 + + + +### 2)JDK8 HashMap线程不安全原因分析 +JDK8 HashMap 不安全,不是因为还会像 JDK7 那样容易成环,而是因为 put、size++、resize、table 发布、链表/红黑树修改都没有加锁或 CAS,多线程并发读写会发生覆盖、丢数据、计数错误和可见性问题。 + + + +> **桶为空时,多线程操作桶,会直接覆盖table[i]** +> + +```plain +if ((p = tab[i = (n - 1) & hash]) == null) + tab[i] = newNode(hash, key, value, null); +``` + +此处最核心原因是操作同一个桶位置,没有加锁,也没有进行CAS,线程不安全。 + + + +> **链表尾插时,p.next 可能互相覆盖** +> + +```java +if ((e = p.next) == null) { + p.next = newNode(hash, key, value, null); + ... + break; +} +``` + +多线程尾插法容易导致p.next正确结果被覆盖。 + + + +> **++size不是原子操作** +> + +```java +if (++size > threshold) + resize(); +``` + +此处++size不是原子操作,会导致最终size结果不准确。 + + + + + +### 3)JDK7 HashMap线程不安全原因分析 +JDK7 并发扩容时,头插法会反转链表,两个线程交叉修改同一批 Entry 的 next 指针,就可能把 A.next 指向 B,同时又把 B.next 指回 A,形成死循环。 + +```plain +T1 线程处理原始链表: + +A -> B -> null + + +T2 线程头插迁移后,把指针改成: + +B -> A -> null + + +T1 线程继续按旧进度迁移,但读到了 T2 线程改过的 B.next: + +B.next = A + + +最后成了循环链表,变成下图: + +A -> B +^ | +|____| +``` + +## getNode() +getNode()方法源码如下 + +```java +final Node getNode(int hash, Object key) { + // tab:底层数组 + // first:桶里的第一个节点 + // e:遍历链表时的当前节点 + // n:数组长度 + // k:临时保存节点 key + Node[] tab; Node first, e; int n; K k; + + // table 不为空、长度大于 0,并且目标桶不为空,才继续查找 + if ((tab = table) != null && (n = tab.length) > 0 && + (first = tab[(n - 1) & hash]) != null) { + + // 先检查桶里的第一个节点 + // 先比 hash,再比 key 引用或 equals + if (first.hash == hash && + ((k = first.key) == key || (key != null && key.equals(k)))) + return first; + + // 第一个节点不是目标 key,并且后面还有节点 + if ((e = first.next) != null) { + + // 如果桶已经树化,走红黑树查找 + if (first instanceof TreeNode) + return ((TreeNode)first).getTreeNode(hash, key); + + // 普通链表,依次向后遍历 + do { + // 找到 hash 和 key 都匹配的节点,直接返回 + if (e.hash == hash && + ((k = e.key) == key || (key != null && key.equals(k)))) + return e; + + // 继续访问下一个节点 + } while ((e = e.next) != null); + } + } + + // table 为空、桶为空,或者遍历完没找到 + return null; +} +``` + + 核心流程: + +1.table 为空,直接返回 null。 + +2.根据 hash 定位桶下标。 + +3.先查桶里的第一个节点。 + +4.如果是红黑树,走树查找。 + +5.否则遍历链表。 + +6.找不到返回 null。 + +## hash() +hash()方法是HashMap中的hash扰动函数,作用是:把 key 的 hashCode() 再处理一下,让高位信息也参与到低位计算,减少哈希冲突。 + +```java +static final int hash(Object key) { + int h; + return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); +} +``` + + 最核心的是这一段:(h = key.hashCode()) ^ (h >>> 16) + +> **为什么要这么做?** +> + +在HashMap中计算桶下标都需要通过:(n - 1) & hash 来计算。又因为n是2的幂,所以这个计算主要以来的是hash的**低位**,高位一直没有利用到。HashMap初始容量为16,则n-1=15,15的二进制位:0000 1111,那么:(n - 1) & hash由于与操作的特性,这实际上只看hash的低4位。**这会导致:如果很多 key 的低位相同,即使高位不同,也会落到同一个桶里。** + +所以 HashMap 做了这个扰动:h ^ (h >>> 16)。把高 16 位右移到低 16 位,再和原 hash 异或,让高位信息参与低位计算。 + +举例: + +```java +>>>是无符号右移:整体向右移动,左边补 0,右边被移出去的低位丢弃。 + +原始 hash: + +高 16 位 低 16 位 +AAAA AAAA BBBB BBBB + +h >>> 16: + +0000 0000 AAAA AAAA + +异或后: + +AAAA AAAA (BBBB BBBB ^ AAAA AAAA) +``` + +## resize() +resize也是HashMap的核心方法之一,源码如下: + +```java +final Node[] resize() { + // 旧数组 + Node[] oldTab = table; + + // 旧容量,table 为空则为 0 + int oldCap = (oldTab == null) ? 0 : oldTab.length; + // 旧扩容阈值 + int oldThr = threshold; + // 新容量、新阈值 + int newCap, newThr = 0; + + // 情况一:旧数组已经存在,说明是正常扩容 + if (oldCap > 0) { + // 已经达到最大容量,不能再扩容 + if (oldCap >= MAXIMUM_CAPACITY) { + threshold = Integer.MAX_VALUE; + return oldTab; + } + // 容量扩大 2 倍 + // 阈值也扩大 2 倍 + else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY && + oldCap >= DEFAULT_INITIAL_CAPACITY) + newThr = oldThr << 1; + } + + // 情况二:数组还没创建,但 threshold > 0 + // 说明构造 HashMap 时指定了初始容量 + // 例如 new HashMap<>(32) + // 此时 threshold 暂时存的是 tableSizeFor(initialCapacity) + else if (oldThr > 0) + newCap = oldThr; + + // 情况三:无参构造 new HashMap<>() + // 第一次 put 时走这里,使用默认容量 16,默认阈值 12 + else { + newCap = DEFAULT_INITIAL_CAPACITY; + newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY); + } + + // 如果上面没有算出新阈值,则按 newCap * loadFactor 计算 + if (newThr == 0) { + float ft = (float)newCap * loadFactor; + newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ? + (int)ft : Integer.MAX_VALUE); + } + + // 更新扩容阈值 + threshold = newThr; + // 创建新数组 + @SuppressWarnings({"rawtypes","unchecked"}) + Node[] newTab = (Node[])new Node[newCap]; + // table 指向新数组 + table = newTab; + + // 如果旧数组不为空,需要迁移旧数据 + if (oldTab != null) { + + // 遍历旧数组每个桶 + for (int j = 0; j < oldCap; ++j) { + Node e; + // 如果当前桶不为空 + if ((e = oldTab[j]) != null) { + + // 旧桶置空,帮助 GC + oldTab[j] = null; + // 情况一:桶里只有一个节点,直接重新计算下标放入新数组 + if (e.next == null) + newTab[e.hash & (newCap - 1)] = e; + // 情况二:桶里是红黑树,走红黑树拆分逻辑 + else if (e instanceof TreeNode) + ((TreeNode)e).split(this, newTab, j, oldCap); + // 情况三:桶里是链表 + else { + // lo 链:扩容后仍然留在原下标 j + Node loHead = null, loTail = null; + // hi 链:扩容后移动到 j + oldCap + Node hiHead = null, hiTail = null; + Node next; + + // 遍历旧链表,把节点拆成 lo 和 hi 两条链 + do { + // 先保存下一个节点 + next = e.next; + // 判断扩容后位置是否不变 + if ((e.hash & oldCap) == 0) { + if (loTail == null) + loHead = e; + else + loTail.next = e; + loTail = e; + } + // 扩容后位置变为 原下标 + oldCap + else { + if (hiTail == null) + hiHead = e; + else + hiTail.next = e; + hiTail = e; + } + + } while ((e = next) != null); + + // lo 链放回原下标 j + if (loTail != null) { + loTail.next = null; + newTab[j] = loHead; + } + // hi 链放到新下标 j + oldCap + if (hiTail != null) { + hiTail.next = null; + newTab[j + oldCap] = hiHead; + } + } + } + } + } + + // 返回新数组 + return newTab; +} +``` + + + + + +# 总结 +HashMap中最核心的概念如下 + +```java +数组 + 链表 + 红黑树 +默认容量 16 +负载因子 0.75 +容量始终是 2 的幂 +链表长度 >= 8 且 table 容量 >= 64 时树化 +树节点过少时退化回链表 +允许 null key / null value +非线程安全 +``` + + + +JDK8和JDK7相比: + +```java +JDK7: +数组 + 链表 + +JDK8: +数组 + 链表 + 红黑树 +``` + diff --git "a/note/JDK/\346\267\261\345\205\245\345\210\206\346\236\220ThreadLocal.md" "b/note/JDK/\346\267\261\345\205\245\345\210\206\346\236\220ThreadLocal.md" index ff6c108..44f8c01 100644 --- "a/note/JDK/\346\267\261\345\205\245\345\210\206\346\236\220ThreadLocal.md" +++ "b/note/JDK/\346\267\261\345\205\245\345\210\206\346\236\220ThreadLocal.md" @@ -150,7 +150,7 @@ Entry是ThreadLocalMap数组中的核心元素,它继承了WeakReference。核 Thread -> ThreadLocalMap -> Entry -> ThreadLocal ``` -在线程池场景下,线程可能长期存活。只要线程不结束,ThreadLocalMap 就还在,Entry 也还在,那么 ThreadLocal 对象就永远无法被 GC。即使业务代码已经不再持有这个 ThreadLocal 变量了,它也会被 Entry 强行引用住。这会导致:**ThreadLocal 对象无法回收,对应的 value 也无法回收。** +在线程池场景下,线程可能长期存活。只要线程不结束,ThreadLocalMap 就还在,Entry 也还在,那么 ThreadLocal 对象就永远无法被 GC。即使业务代码已经不再持有这个 ThreadLocal 变量了,它也会被 Entry 强行引用住。这会导致:**ThreadLocal 对象无法回收,对应的 value 也无法回收。** 这个后果就是线程长期持有已经没用的 ThreadLocal 和 value,导致内存释放不了,严重时内存泄漏、数据串用、甚至 OOM。 @@ -516,7 +516,9 @@ new Thread(() -> { }).start(); ``` - 而 InheritableThreadLocal 可以让子线程继承父线程的值: +```plain +而 InheritableThreadLocal 可以让子线程继承父线程的值: +``` ```java InheritableThreadLocal local = new InheritableThreadLocal<>(); @@ -576,7 +578,7 @@ private ThreadLocalMap(ThreadLocalMap parentMap) { -但是现在基本都没有直接通过new Thread()的方式创建线程了,基本都是通过线程池来管理线程。而在常规业务线程池里,InheritableThreadLocal 基本不适合作为上下文传递方案。它的继承时机是**“创建线程时”**,而线程池的线程通常早就创建好了,任务提交时不会重新继承父线程上下文。 +但是现在基本都没有直接通过new Thread()的方式创建线程了,基本都是通过线程池来管理线程。而在常规业务线程池里,InheritableThreadLocal 基本不适合作为上下文传递方案。它的继承时机是**“创建线程时”**,而线程池的线程通常早就创建好了,任务提交时不会重新继承父线程上下文。 线程池上下文传递方案,用的最多的就是阿里的TransmittableThreadLocal,简称 TTL。 @@ -620,7 +622,139 @@ class TtlRunnable implements Runnable { } ``` - + 总结:TransmittableThreadLocal在线程池里传值,是通过包装任务,在任务提交时捕获父线程的 TTL 快照,在工作线程执行前恢复这份快照,执行结束后再还原工作线程原上下文来实现的。 +# ThreadLocal内存泄漏代码分析 +下面是一段ThreadLocal内存泄漏的伪代码,通过这段伪代码加深ThreadLocal底层原理的理解。 + +```java +static final ThreadLocal USER_CONTEXT = new ThreadLocal<>(); + +void handleRequest(Request request) { + UserInfo userInfo = getUserInfo(request); + USER_CONTEXT.set(userInfo); + + doBusiness(); + + // 忘记执行 USER_CONTEXT.remove() +} +``` + +上述这段代码是用户登录之后,获取用户信息并存到ThreadLocal中,但是方法结束后并未执行:USER_CONTEXT.remove()移除ThreadLocal中的用户信息,这会造成内存泄漏,下面通过引用链来分析下内存泄漏的原因。 + +方法在USER_CONTEXT.set(userInfo)之后,引用关系如下图: + +```java +GC Roots +│ +├── ClassLoader +│ │ 强引用 +│ ▼ +│ Class对象 +│ │ 静态字段强引用 +│ ▼ +│ USER_CONTEXT +│ │ 强引用 +│ ▼ +│ ThreadLocal对象 ◀-------------------┐ +│ │ +└── 工作线程 Thread │ + │ │ + ├── 线程栈 │ + │ │ │ + │ ▼ │ + │ handleRequest()栈帧 │ + │ │ │ + │ ▼ │ + │ 局部变量 userInfo │ + │ │ 强引用 │ + │ ▼ │ + │ UserInfo对象 ◀──────────┐ │ + │ │ │ + └── threadLocals │ │ + │ 强引用 │ │ + ▼ │ │ + ThreadLocalMap │ │ + │ 强引用 │ │ + ▼ │ │ + Entry │ │ + ├── value强引用 ─────┘ │ + │ │ + └---- key弱引用 -----------┘ +``` + +这里有个细节,调用了USER_CONTEXT.set(userInfo)之后,ThreadLocal对象被USER_CONTEXT强引用引用,同时还被Entry这个弱引用给引用了。 + + + +当handleRequest()方法调用完之后,userInfo局部变量消失,此时引用关系图如下 + +```java +GC Roots +│ +├── ClassLoader +│ │ 强引用 +│ ▼ +│ Class对象 +│ │ 静态字段强引用 +│ ▼ +│ USER_CONTEXT +│ │ 强引用 +│ ▼ +│ ThreadLocal对象 ◀--------------------┐ +│ │ +└── 工作线程 Thread │ + │ │ + ├── 线程栈 │ + │ │ │ + │ └── handleRequest栈帧已消失 │ + │ │ + └── threadLocals │ + │ 强引用 │ + ▼ │ + ThreadLocalMap │ + │ 强引用 │ + ▼ │ + Entry │ + ├---- key弱引用 -----------┘ + │ + └── value强引用 + │ + ▼ + UserInfo对象 +``` + +此时发生变化的是,局部变量对应的引用已经断开。 + +```java +局部变量userInfo ──X──> UserInfo对象 +``` + + + +因此,即使局部变量消失,`UserInfo` 仍然无法被 GC 回收。只有执行 `USER_CONTEXT.remove()` 清理对应 Entry,才能断开这条引用链。 + + + +所以正确的写法如下 + +```java +private static final ThreadLocal USER_CONTEXT = + new ThreadLocal<>(); + +public void handleRequest(Request request) { + try { + UserInfo userInfo = getUserInfo(request); + USER_CONTEXT.set(userInfo); + + // 处理具体业务 + doBusiness(); + } finally { + // 无论正常结束还是发生异常,都必须清理 + USER_CONTEXT.remove(); + } +} +``` + diff --git "a/note/JDK/\346\267\261\345\205\245\350\247\243\346\236\220ThreadPoolExecutor\345\272\225\345\261\202\345\216\237\347\220\206.md" "b/note/JDK/\346\267\261\345\205\245\350\247\243\346\236\220ThreadPoolExecutor\345\272\225\345\261\202\345\216\237\347\220\206.md" index ba9b143..8f55d40 100644 --- "a/note/JDK/\346\267\261\345\205\245\350\247\243\346\236\220ThreadPoolExecutor\345\272\225\345\261\202\345\216\237\347\220\206.md" +++ "b/note/JDK/\346\267\261\345\205\245\350\247\243\346\236\220ThreadPoolExecutor\345\272\225\345\261\202\345\216\237\347\220\206.md" @@ -11,7 +11,7 @@ private final AtomicInteger ctl = new AtomicInteger(ctlOf(RUNNING, 0)); -在不同操作系统下,Java 中的 Integer 变量都是32位,ThreadPoolExecutor 使用前3位(31~29)表示线程池状态,用后29位(28~0)表示活跃线程数。 +在不同操作系统下,Java 中的 Integer 变量都是32位,ThreadPoolExecutor 使用前3位(31 ~ 29)表示线程池状态,用后29位(28 ~ 0)表示活跃线程数。 ## COUNT_BITS COUNT_BITS的作用:用来划分 ctl 这个 int 变量的高低位边界。 @@ -22,7 +22,7 @@ private static final int COUNT_BITS = Integer.SIZE - 3; // 29 -int的最大位值是32位,32 - 3 = 29。通过这29来控制高3位存线程池状态**(注意这里不是线程,是线程池)**,低29位表示线程池中worker数量。通过位移来实现高效操作。 +int的最大位值是32位,32 - 3 = 29。通过这29来控制高3位存线程池状态(注意这里不是线程,是线程池),低29位表示线程池中worker数量。通过位移来实现高效操作。 ## CAPACITY CAPACITY的作用:用来表示 workerCount 线程数量部分的最大容量,同时也作为低 29 位的**掩码**。 @@ -40,11 +40,10 @@ private static final int CAPACITY = (1 << COUNT_BITS) - 1; 2)对左移结果 - 1 -(1 << COUNT_BITS) - 1:就是00100000 00000000 00000000 00000000 - 1,对于二进制,如果高位减1,会借位到对应位置-1。举例如下: +(1 << COUNT_BITS) - 1:就是00100000 00000000 00000000 00000000 - 1,对于二进制,如果高位减1,会借位到对应位置减1。举例如下: + 01000 - 1 = 01000 - 01111 = 00111 + 00100000 00000000 00000000 00000000 - 1 = 00100000 00000000 00000000 00000000 - 00111111 11111111 11111111 11111111 = 00011111 11111111 11111111 11111111 -+ 口诀:对于这种数:00100000 00000000 00000000 00000000,它是“某一位是 1,右边全是 0”。 ### 【扩展】掩码 掩码 Mask,就是一串二进制位,用来通过位运算“筛选”出你想要的部分。在ThreadPoolExecutor里: diff --git "a/note/kafka/Kafka ISR \345\272\225\345\261\202\345\216\237\347\220\206.md" "b/note/kafka/Kafka ISR \345\272\225\345\261\202\345\216\237\347\220\206.md" index 02e5bd2..93a9b36 100644 --- "a/note/kafka/Kafka ISR \345\272\225\345\261\202\345\216\237\347\220\206.md" +++ "b/note/kafka/Kafka ISR \345\272\225\345\261\202\345\216\237\347\220\206.md" @@ -1,6 +1,10 @@ + 当前分析版本是kafka最新版本(版本随时变化,最新分析代码请关注仓库:https://github.com/coderbruis/kafka source_code_analysis分支,底层原理持续更新) + 转载请标明出处 +# Kafka HW,ISR,LEO关系图 + +![Kafka_HW_ISR_LEO](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_HW_ISR_LEO.png) + # Kafka ISR是什么?解决什么问题? ## 是什么? Kafka ISR是In-Sync Replicas,意思是“与leader保持同步的副本集合”。在Kafka中会有leader副本和follower副本,下面举例: diff --git "a/note/kafka/kafka rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" "b/note/kafka/kafka Rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" similarity index 81% rename from "note/kafka/kafka rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" rename to "note/kafka/kafka Rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" index 409bc93..bf3bde3 100644 --- "a/note/kafka/kafka rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" +++ "b/note/kafka/kafka Rebalance\346\240\270\345\277\203\351\200\273\350\276\221\345\210\206\346\236\220.md" @@ -1,6 +1,12 @@ + 当前分析版本是kafka最新版本(版本随时变化,最新分析代码请关注仓库:[https://github.com/coderbruis/kafka](https://github.com/coderbruis/kafka) **source_code_analysis分支**,底层原理持续更新) + 转载请标明出处 +# Kafka Rebalance流程图 + +![Kafka EAGER Rebalance](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_EAGER_rebalance.png) + +![Kafka Cooperative Rebalance](https://github.com/coderbruis/JavaSourceCodeLearning/releases/download/images-v1/kafka_COOPERATIVE_rebalance.png) + # Kafka Rebalance 核心流程 `KafkaConsumer.poll()` 是消费者触发 rebalance 的主要入口。 @@ -629,20 +635,47 @@ if (protocol == RebalanceProtocol.COOPERATIVE && # EAGER 和 COOPERATIVE 的差异 ## EAGER -EAGER rebalance 的特点是简单直接。 +EAGER核心特点:全量停、全量分、再恢复 + +### EAGER Rebalance流程 + +第一步:触发 Rebalance +1. 成员变化:join / leave / session timeout / max.poll.interval 超时等。 +2. 订阅或元数据变化:订阅 topic 变化、正则订阅匹配变化、partition 数变化等。 + +第二步:Coordinator 进入 Rebalance 状态 +1. Coordinator 将 group 状态切到 PreparingRebalance(可能不是第一次进入rebalance,所以这里状态已经是PreparingRebalance了,正常第一次是在JoinGroup进入Coordinator里,会将状态变更为PreparingRebalance。) +2. 现有成员会通过 heartbeat 或 poll 流程感知需要重新加入 group。 +3. 新成员或需要重分配的成员准备发送 JoinGroupRequest。 + +第三步:EAGER 全员撤销旧 assignment,然后 JoinGroup +1. 每个 consumer 在 JoinGroup 前执行 onJoinPrepare。 +2. EAGER 协议下 revoke 当前持有的所有 partitions。 +3. 调用 onPartitionsRevoked(allAssignedPartitions)。 +4. 清空本地 assignment,停止这些 partitions 的消费。全组成员进入消费暂停状态(STW)。 +5. 向 coordinator 发送 JoinGroupRequest,携带 subscription 和支持的 assignor。 + +第四步:JoinGroup 完成成员协商 +1. Coordinator 收集本轮成员的 JoinGroupRequest。 +2. 选择 leader consumer。 +3. 选择 assignment strategy / protocol,leader consumer生成新的assignment。 +4. 生成新的 generationId,同时将group状态变更为COMPLETING_REBALANCE。 +5. JoinGroupResponse 返回给成员。 +6. Leader 会拿到所有成员的 subscription metadata。 + +第五步:分配 + SyncGroup +1. Leader consumer 执行分配算法,比如 Range / Sticky。 +2. Leader 通过 SyncGroupRequest 把全组 assignment 发给 coordinator。 +3. Follower 也发送 SyncGroupRequest,但通常不带 assignment。 +4. Coordinator 保存本轮 assignment,group状态变更为STABLE。 +5. Coordinator 通过各自的 SyncGroupResponse 返回每个 consumer 自己的 assignment。 + +第六步:恢复消费 +1. Consumer 收到自己的 assignment。 +2. 更新本地 assignment。 +3. 调用 onPartitionsAssigned(newAssignedPartitions)。 +4. 从对应 offset 开始拉取消息,恢复消费。 -```plain -onJoinPrepare: - revoke all partitions - clear local assignment - -leader assign: - assign all partitions again - -onJoinComplete: - assign new partitions - trigger onPartitionsAssigned -``` 优点: @@ -655,23 +688,70 @@ onJoinComplete: + 即使某些 partition 仍然分给同一个 consumer,也会先 revoke 再 assign。 ## COOPERATIVE -COOPERATIVE rebalance 的特点是渐进迁移。 - -```plain -onJoinPrepare: - keep still-owned partitions - revoke only obviously invalid partitions +COOPERATIVE rebalance 的特点是渐进迁移,不会撤销所有分区导致全部分区STW。 + +### COOPERATIVE Rebalance流程 + +第一轮:标记迁移,旧 owner revoke(onwer表示持有parition的consumer) +1. 成员变化、订阅变化或元数据变化触发 rebalance。 +2. consumer 发送 JoinGroup 给 coordinator。 + JoinGroup metadata 里包含: + ○ 当前 subscription + ○ 当前本地已分配分区,也就是 ownedPartitions +3. coordinator 收集所有成员的 JoinGroup。 + coordinator 负责: + ○ 选 leader + ○ 选 assignor/protocol + ○ 把所有成员 subscription metadata 返回给 leader + coordinator收集完所有的JoinGroup请求之后,会进入新的generationId(递增)。 +4. leader consumer 执行 assignor。 + CooperativeStickyAssignor 会: + ○ 先计算目标 assignment + ○ 如果某个分区要从旧 owner 转给新 owner + ○ 但旧 owner 在本轮 JoinGroup 里仍上报该分区为 owned + ○ 那么本轮不会把该分区分给新 owner + ○ 同时旧 owner 的本轮 assignment 不再包含该分区 +5. leader Consumer 通过 SyncGroup 把全组 assignment 提交给 coordinator。其他consumer也会发送一个空的SyncGroup给coorinator +6. coordinator 保存 assignment,并通过 SyncGroupResponse 返回每个 consumer 自己的 assignment。 +7. consumer 处理 SyncGroupResponse。 + 本地计算: +```text + owned = 当前本地 assignment + assigned = 本轮收到的 assignment + + revoked = owned - assigned + added = assigned - owned +``` + +需要注意,第一轮主要是做撤销,但是也可能直接添加新的分区分配结果。因为如果这一轮有某个分区本来就没有旧onwer(旧的持有这个parition的consumer),这一轮就可以直接分配给新onwer。 +8. 如果 revoked 非空: + ○ 调用 onPartitionsRevoked(revoked) + ○ 调用 requestRejoin() + ○ 后续把本地 assignment 更新为 assigned +9. 如果 added 非空: + ○ 调用 onPartitionsAssigned(added) + 注意:正在从旧 owner 转移给新 owner 的分区,第一轮不会出现在新 owner 的 added 里。 + 第二轮:旧owner撤销分区,新owner获得第一轮撤销的分区 +1. 因为第一轮有 consumer 调用了 requestRejoin(),group 再次 rebalance。 +2. consumer 再次发送 JoinGroup。 + 此时旧 owner 的本地 assignment 已经更新,所以它上报的 ownedPartitions 不再包含刚刚 revoked 的分区。 + 被撤销的分区不会通过JoinGroup发送给Coordinator,revokeList和addLIst都是通过集合计算来得出的。 + coordinator收集完所有的JoinGroup请求之后,会进入新的generationId(递增)。 +3. coordinator 再次收集成员,选 leader,选协议。 + 然后将所有成员 subscription metadata 返回给 leader consumer。 +4. leader 再次执行 assignor。 + 这次 assignor 发现: + ○ 该分区已经没有旧 owner 上报 owned + ○ 可以安全分配给新 owner +5. leader 通过 SyncGroup 提交新的全组 assignment。 +6. coordinator 通过 SyncGroupResponse 返回各成员自己的 assignment。 +7. 新 owner 处理 assignment。 + 本地计算: + added = assigned - owned +8. 新 owner 对新增分区调用: + onPartitionsAssigned(added) -leader assign: - do not immediately reassign partitions still owned by others -onJoinComplete: - revoke partitions that need migration - request another rejoin if needed - -next rebalance: - assign released partitions to new owners -``` 优点: diff --git "a/note/kafka/kafka\346\234\200\346\226\260\347\211\210\346\234\254\346\240\270\345\277\203\346\246\202\345\277\265\344\270\216\346\240\270\345\277\203\345\216\237\347\220\206\346\200\273\347\273\223.md" "b/note/kafka/kafka\346\234\200\346\226\260\347\211\210\346\234\254\346\240\270\345\277\203\346\246\202\345\277\265\344\270\216\346\240\270\345\277\203\345\216\237\347\220\206\346\200\273\347\273\223.md" new file mode 100644 index 0000000..8a4052f --- /dev/null +++ "b/note/kafka/kafka\346\234\200\346\226\260\347\211\210\346\234\254\346\240\270\345\277\203\346\246\202\345\277\265\344\270\216\346\240\270\345\277\203\345\216\237\347\220\206\346\200\273\347\273\223.md" @@ -0,0 +1,1865 @@ ++ 转载请标明出处 + + + +本章旨在快速扫盲\回顾 Apache Kafka 最新版本最核心、最重要的概念以及原理,不做具体框架源码级深入分析。 + +截至本文编写时间,Apache Kafka 官网下载页显示最新支持版本为 **Kafka 4.3.1**,发布日期为 **2026-06-25**。Kafka 4.x 之后的核心变化是 **ZooKeeper 架构退出主线,KRaft 成为 Kafka 元数据管理和集群控制的核心模式**。 + +参考来源: + ++ Apache Kafka Downloads: https://kafka.apache.org/community/downloads/ ++ Apache Kafka 4.3 Documentation: https://kafka.apache.org/43/ ++ Apache Kafka Documentation: https://kafka.apache.org/documentation/ + +重点覆盖: + ++ Kafka 是什么 ++ Topic、Partition、Replica、Broker ++ Producer、Consumer、Consumer Group ++ Offset、Log、Segment、Index ++ ISR、Leader、Follower、高水位 HW ++ Controller、KRaft、Metadata Quorum ++ 生产者发送流程、ACK、幂等、事务 ++ 消费者拉取流程、位移提交、Rebalance ++ 传统 Consumer Group 和新版 Consumer 协议 ++ Share Group / Queues for Kafka ++ 顺序性、可靠性、可用性、一致性 ++ Page Cache、顺序写、零拷贝 ++ Log Retention、Log Compaction ++ Tiered Storage ++ Kafka Connect、Kafka Streams ++ 安全机制、监控指标、常见问题排查 ++ 工程选型和实践建议 + +# Kafka +Kafka 是 **一个分布式事件流平台,用于高吞吐、低延迟、可持久化地发布、存储、订阅和处理事件数据**。 + +它主要解决: + +1. **系统解耦** +生产者只把消息写入 Kafka,不需要直接调用所有下游系统。 +2. **削峰填谷** +流量高峰先写入 Kafka,下游按自身能力消费。 +3. **数据持久化** +消息不是消费后立即删除,而是按保留策略存储一段时间。 +4. **实时数据流处理** +适合日志、埋点、监控、交易事件、CDC、实时计算。 +5. **多订阅方消费** +同一份数据可以被多个消费者组独立消费。 + +**传统同步调用** + +```plain +订单服务 + -> 库存服务 + -> 积分服务 + -> 风控服务 + -> 推荐服务 +``` + +调用链长,任何下游慢都会影响上游。 + +**使用 Kafka** + +```plain +订单服务 + -> 写入 order_created Topic + +库存服务消费 +积分服务消费 +风控服务消费 +推荐服务消费 +``` + +各系统通过事件解耦。 + +**需要注意** + ++ Kafka 不是传统意义上“发完即删”的简单消息队列。 ++ Kafka 的核心抽象是分布式日志。 ++ Kafka 强在高吞吐、可扩展、可回放、多消费者订阅。 ++ Kafka 不适合所有低延迟 RPC 场景,也不替代数据库事务。 ++ Kafka 4.x 新集群应优先理解 KRaft,而不是旧 ZooKeeper 架构。 + +一句话总结: + +**Kafka 是以分布式日志为核心的事件流平台,通过 Topic、Partition、Offset 和 Consumer Group 实现高吞吐、可持久化、可回放的数据流。** + +# Topic +Topic 是 **Kafka 中消息的逻辑分类,相当于一类事件流的名字**。 + +它主要解决: + +1. **按业务分类消息** +2. **隔离不同事件流** +3. **让生产者和消费者通过名称解耦** +4. **作为权限、保留策略、分区配置的管理单位** + +**例子** + +```plain +order_created +payment_success +user_login +stock_changed +app_log +``` + +**生产和消费** + +```plain +Producer -> order_created Topic +Consumer <- order_created Topic +``` + +**Topic 不是物理上的一个文件** + +Kafka 内部会把 Topic 拆成多个 Partition: + +```plain +order_created + -> partition-0 + -> partition-1 + -> partition-2 +``` + +**需要注意** + ++ Topic 命名要稳定,避免随意变更。 ++ Topic 数量过多会增加元数据和运维成本。 ++ Topic 的分区数、保留时间、清理策略要结合业务设计。 ++ 不同业务语义的数据不要随意混入同一个 Topic。 + +一句话总结: + +**Topic 是 Kafka 消息的逻辑分类,真正承载数据和并行能力的是 Topic 下的 Partition。** + +# Partition +Partition 是 **Topic 的物理分片,也是 Kafka 并行、顺序和扩展能力的基础单位**。 + +它主要解决: + +1. **单个 Topic 水平扩展** +2. **提升读写吞吐** +3. **让消费者并行消费** +4. **保证分区内消息顺序** + +**结构** + +```plain +Topic: order_created + +partition-0: + msg0, msg1, msg2 + +partition-1: + msg0, msg1, msg2 + +partition-2: + msg0, msg1, msg2 +``` + +**分区内有序** + +```plain +partition-0: + offset 0 + offset 1 + offset 2 +``` + +Kafka 只保证: + +```plain +同一个 Partition 内有序 +``` + +不保证: + +```plain +整个 Topic 全局有序 +``` + +**分区选择** + +```plain +有 key: + 根据 key hash 选择分区 + +无 key: + 按生产者分区策略分配 + +自定义: + 业务实现 Partitioner +``` + +**需要注意** + ++ 分区数决定消费者组内最大并行度。 ++ 分区数不是越多越好,过多会增加文件、网络和元数据成本。 ++ 扩容分区会影响 key 到分区的映射,可能破坏同 key 顺序。 ++ 需要同一业务实体有序时,应让同一 key 进入同一分区。 + +一句话总结: + +**Partition 是 Kafka 并行和顺序的核心单位,Kafka 保证分区内有序,不保证 Topic 全局有序。** + +# Broker +Broker 是 **Kafka 集群中的服务节点,负责接收生产请求、存储分区日志、处理消费请求和复制数据**。 + +它主要解决: + +1. **承载 Topic Partition 数据** +2. **处理客户端读写请求** +3. **参与副本复制** +4. **向集群汇报状态** +5. **作为 KRaft 节点参与元数据管理,取决于角色配置** + +**集群结构** + +```plain +Broker-1 + -> topicA partition-0 leader + -> topicB partition-1 follower + +Broker-2 + -> topicA partition-1 leader + -> topicA partition-0 follower + +Broker-3 + -> topicA partition-2 leader + -> topicB partition-0 follower +``` + +**Broker 角色** + +```plain +普通 Broker: + 处理数据读写 + +Controller: + 管理集群元数据和分区状态 + +Combined 节点: + 同时承担 Broker 和 Controller 角色 +``` + +**需要注意** + ++ Broker 宕机会触发副本 Leader 切换。 ++ 单个 Broker 磁盘、网络、CPU 都可能成为瓶颈。 ++ Broker 不是无状态服务,磁盘数据和 broker.id/node.id 很关键。 ++ 生产环境要合理规划磁盘、网络、机架和副本分布。 + +一句话总结: + +**Broker 是 Kafka 集群的数据服务节点,负责消息读写、日志存储、副本复制,并在 KRaft 架构中配合集群元数据管理。** + +# Replica +Replica 是 **Partition 的副本,用于提升 Kafka 数据可靠性和可用性**。 + +它主要解决: + +1. **Broker 宕机后的数据可用** +2. **降低单点故障风险** +3. **支持 Leader 故障切换** +4. **提升数据持久性** + +**副本结构** + +```plain +partition-0 + leader: Broker-1 + follower: Broker-2 + follower: Broker-3 +``` + +**Leader 副本** + +```plain +处理生产者写入 +处理消费者读取 +维护分区日志主流程 +``` + +**Follower 副本** + +```plain +从 Leader 拉取数据 +保持日志同步 +在 Leader 故障时可能被选为新 Leader +``` + +**副本因子** + +```plain +replication.factor = 3 +``` + +表示每个 Partition 有 3 个副本。 + +**需要注意** + ++ 副本越多,可靠性越好,但存储和复制成本越高。 ++ 生产常见副本因子是 3。 ++ Follower 默认不对普通消费者提供读服务。 ++ 副本分布要避免多个副本落在同一故障域。 + +一句话总结: + +**Replica 通过 Leader/Follower 副本复制提升 Kafka 的可靠性和可用性,生产环境通常使用 3 副本。** + +# ISR +ISR 是 **In-Sync Replicas,同步副本集合,表示当前跟得上 Leader 的副本列表**。 + +它主要解决: + +1. **判断哪些副本是健康同步副本** +2. **控制消息提交安全性** +3. **Leader 故障时选择可靠新 Leader** +4. **配合 acks 和 min.insync.replicas 保证写入可靠性** + +**结构** + +```plain +partition-0 + leader: Broker-1 + ISR: [Broker-1, Broker-2, Broker-3] +``` + +如果 Broker-3 落后太多: + +```plain +ISR: [Broker-1, Broker-2] +``` + +**写入确认** + +```plain +acks=all +min.insync.replicas=2 +``` + +含义: + +```plain +消息至少要被 ISR 中足够数量副本确认 +才认为写入成功 +``` + +**需要注意** + ++ ISR 不是固定副本列表,会动态变化。 ++ ISR 缩小通常表示副本复制延迟或故障。 ++ min.insync.replicas 设置过低会降低可靠性。 ++ 设置过高会降低可用性,副本不足时写入失败。 + +一句话总结: + +**ISR 表示当前与 Leader 保持同步的副本集合,是 Kafka 判断写入可靠性和故障切换安全性的核心机制。** + +# Offset +Offset 是 **消息在 Partition 内的递增位置编号**。 + +它主要解决: + +1. **标识消息位置** +2. **支持消费者断点续读** +3. **支持消息回放** +4. **让不同消费者组独立维护消费进度** + +**结构** + +```plain +partition-0: + offset 0 -> msgA + offset 1 -> msgB + offset 2 -> msgC +``` + +**消费者进度** + +```plain +consumer group A: + partition-0 committed offset = 2 + +consumer group B: + partition-0 committed offset = 0 +``` + +同一个 Topic 可以被多个消费者组独立消费。 + +**Offset 提交** + +```plain +消费消息 + -> 处理业务 + -> 提交 offset +``` + +**需要注意** + ++ Offset 只在单个 Partition 内有意义。 ++ 提交 offset 表示消费者认为之前的数据已经处理完成。 ++ 先提交再处理可能丢消息。 ++ 先处理再提交可能重复消费。 ++ Kafka 通常要求业务端做好幂等处理。 + +一句话总结: + +**Offset 是 Partition 内消息位置,消费者通过提交 Offset 记录消费进度,从而支持断点续读和消息回放。** + +# Log +Kafka Log 是 **Partition 在磁盘上的追加写日志,每条消息按 Offset 顺序追加到日志末尾**。 + +它主要解决: + +1. **消息持久化** +2. **顺序写入** +3. **按 Offset 查询** +4. **支持消息保留和回放** + +**日志结构** + +```plain +topic-partition/ + 00000000000000000000.log + 00000000000000000000.index + 00000000000000000000.timeindex + 00000000000000100000.log + 00000000000000100000.index + 00000000000000100000.timeindex +``` + +**追加写** + +```plain +Producer 写入 + -> Broker 追加到当前 active segment + -> 分配 offset + -> 等待确认 +``` + +**读取** + +```plain +Consumer 请求 offset + -> Broker 通过 index 定位 segment + -> 从 log 文件顺序读取 + -> 返回消息批次 +``` + +**需要注意** + ++ Kafka 不是随机更新消息,而是追加日志。 ++ 日志文件按 Segment 切分,便于删除、压缩和索引。 ++ 磁盘顺序写是 Kafka 高吞吐的重要原因。 ++ 日志保留时间到期后,旧消息会被清理,不能无限回放。 + +一句话总结: + +**Kafka 的核心存储模型是 Partition 追加日志,消息按 Offset 顺序写入磁盘,并通过 Segment 和索引支持高效读写。** + +# Segment 和 Index +Segment 是 **Kafka 把 Partition 日志切成多个文件段的机制**。 + +Index 是 **Kafka 用于快速定位 Offset 或时间戳对应日志位置的稀疏索引**。 + +它主要解决: + +1. **避免单个日志文件过大** +2. **方便按时间或大小清理旧数据** +3. **提升 Offset 定位效率** +4. **支持快速查找时间戳附近消息** + +**Segment 文件** + +```plain +00000000000000000000.log +00000000000000000000.index +00000000000000000000.timeindex +``` + +文件名前缀表示: + +```plain +该 Segment 起始 offset +``` + +**索引定位** + +```plain +Consumer 请求 offset=12345 + -> 找到起始 offset <= 12345 的 segment + -> 查 offset index 找近似位置 + -> 从 log 文件顺序扫描到目标消息 +``` + +**需要注意** + ++ Kafka 使用稀疏索引,不是每条消息都有索引。 ++ Segment 越大,文件数量少,但清理粒度更粗。 ++ Segment 越小,清理更灵活,但文件数量更多。 ++ 索引损坏通常可以根据 log 文件重建。 + +一句话总结: + +**Segment 把分区日志拆成可管理文件段,Index 用稀疏索引快速定位消息位置,是 Kafka 高效存储和清理的基础。** + +# Producer +Producer 是 **向 Kafka Topic 写入消息的客户端**。 + +它主要解决: + +1. **序列化业务数据** +2. **选择目标 Topic 和 Partition** +3. **批量发送消息** +4. **处理重试、幂等和事务** +5. **接收 Broker 写入确认** + +**发送流程** + +```plain +业务调用 send() + -> 序列化 key/value/header + -> 分区器选择 partition + -> 写入 RecordAccumulator + -> Sender 线程批量发送 + -> Broker 写入日志 + -> 返回 ack +``` + +**重要参数** + +```plain +acks: + 写入确认级别 + +batch.size: + 批次大小 + +linger.ms: + 等待聚合批次时间 + +compression.type: + 压缩方式 + +retries: + 重试次数 + +enable.idempotence: + 幂等生产 +``` + +**需要注意** + ++ batch.size 和 linger.ms 影响吞吐和延迟。 ++ 压缩可以降低网络和磁盘成本,但增加 CPU 开销。 ++ 重试可能导致乱序,幂等生产可以降低重复写入风险。 ++ 生产者发送成功不等于所有消费者已经消费。 + +一句话总结: + +**Producer 通过序列化、分区、批量、压缩、重试和 ACK 机制把业务事件高效可靠地写入 Kafka。** + +# ACK +ACK 是 **生产者写入消息时要求 Broker 返回确认的级别**。 + +它主要解决: + +1. **控制写入可靠性** +2. **平衡吞吐和数据安全** +3. **配合副本同步策略** + +**acks=0** + +```plain +Producer 不等待 Broker 确认 +吞吐高 +可能丢消息 +``` + +**acks=1** + +```plain +Leader 写入成功后返回 +性能较好 +Leader 宕机且 Follower 未同步时可能丢消息 +``` + +**acks=all** + +```plain +Leader 等待 ISR 副本满足要求后返回 +可靠性最高 +延迟更高 +``` + +**常见可靠配置** + +```plain +acks=all +min.insync.replicas=2 +replication.factor=3 +enable.idempotence=true +``` + +**需要注意** + ++ acks=all 还需要合理设置 min.insync.replicas。 ++ min.insync.replicas 过低会降低副本确认意义。 ++ 副本不足时,acks=all 可能导致写入失败。 ++ 高可靠配置会牺牲部分可用性和延迟。 + +一句话总结: + +**ACK 决定生产者等待到什么程度才认为写入成功,acks=all 配合 ISR 和 min.insync.replicas 是高可靠写入的核心。** + +# 幂等生产者 +幂等生产者是 **Kafka 通过 Producer ID、Producer Epoch 和 Sequence Number 避免生产者重试导致重复写入的机制**。 + +它主要解决: + +1. **网络超时后重试可能重复写入** +2. **Broker 已写入但 ACK 丢失** +3. **单分区内重试乱序和重复** + +**问题场景** + +```plain +Producer 发送 msg-1 + -> Broker 写入成功 + -> ACK 在网络中丢失 + -> Producer 重试 msg-1 + -> 可能重复写入 +``` + +**幂等机制** + +```plain +每个 Producer 有 PID +每个分区维护递增 sequence +Broker 检查 sequence 是否连续 +重复 sequence 被识别并丢弃 +``` + +**需要注意** + ++ 幂等生产者主要保证单 Producer 会话内、单分区的幂等写入。 ++ 它不等于业务幂等。 ++ 应用重启、业务重放、消费者重复处理仍要业务幂等。 ++ 跨分区、跨 Topic 的原子写入需要事务。 + +一句话总结: + +**幂等生产者通过 PID 和序列号避免发送重试造成的重复写入,是 Kafka 精确一次语义的重要基础。** + +# Kafka 事务 +Kafka 事务是 **让生产者把多条消息、多个分区写入和消费位移提交作为一个原子单元提交或回滚的机制**。 + +它主要解决: + +1. **跨分区原子写入** +2. **消费-处理-生产链路一致性** +3. **Exactly Once 语义基础** +4. **避免下游读到未提交事务消息** + +**典型流程** + +```plain +initTransactions +beginTransaction + -> consume input records + -> produce output records + -> sendOffsetsToTransaction +commitTransaction +``` + +失败时: + +```plain +abortTransaction +``` + +**读隔离级别** + +```plain +read_uncommitted: + 可以读到未提交事务消息 + +read_committed: + 只读已提交事务消息 +``` + +**需要注意** + ++ Kafka 事务解决 Kafka 内部读写链路的一致性,不自动保证外部数据库事务。 ++ 事务会增加协调成本和延迟。 ++ transactional.id 必须稳定且唯一。 ++ EOS 仍然要求业务端正确处理外部副作用。 + +一句话总结: + +**Kafka 事务把多分区写入和消费位移提交纳入同一原子单元,是 Kafka Exactly Once 处理链路的核心机制。** + +# Consumer +Consumer 是 **从 Kafka Topic 拉取并处理消息的客户端**。 + +它主要解决: + +1. **按 Offset 拉取消息** +2. **维护消费进度** +3. **参与消费者组分区分配** +4. **处理重试、提交和回放** + +**消费流程** + +```plain +Consumer 加入 Consumer Group + -> 获取分区分配 + -> poll 拉取消息 + -> 执行业务处理 + -> 提交 offset +``` + +**拉模式** + +Kafka 消费是客户端主动拉取: + +```plain +Consumer -> Broker: fetch request +Broker -> Consumer: records +``` + +优点: + +```plain +消费者按自身能力拉取 +容易做批量 +方便控制进度 +``` + +**需要注意** + ++ poll 要持续调用,否则可能被认为消费者失活。 ++ 消费处理慢会导致消费堆积。 ++ 自动提交 offset 简单但容易产生丢失或重复风险。 ++ 业务处理必须考虑重复消费。 + +一句话总结: + +**Consumer 通过 poll 主动拉取消息,并用 Offset 管理消费进度;消费可靠性核心在于处理和提交 Offset 的顺序。** + +# Consumer Group +Consumer Group 是 **Kafka 实现消费负载均衡和多订阅方隔离的机制**。 + +它主要解决: + +1. **同组消费者分摊分区消费** +2. **不同组独立消费同一 Topic** +3. **消费者故障后自动接管分区** +4. **支持水平扩展消费能力** + +**同组消费** + +```plain +Topic: order_created + partition-0 + partition-1 + partition-2 + +Group-A: + consumer-1 -> partition-0 + consumer-2 -> partition-1 + consumer-3 -> partition-2 +``` + +同一个消费者组内: + +```plain +一个 Partition 同一时刻通常只分配给一个 Consumer +``` + +不同消费者组: + +```plain +Group-A 可以消费一遍 +Group-B 也可以独立消费一遍 +``` + +**需要注意** + ++ 同一组消费者数量超过分区数时,多出来的消费者会空闲。 ++ 分区数决定同组最大并行消费数。 ++ 消费者上下线会触发 Rebalance。 ++ 不同业务系统通常应该使用不同 group.id。 + +一句话总结: + +**Consumer Group 让同一组消费者分摊分区消费,让不同组可以独立订阅同一份 Kafka 数据。** + +# Offset 提交 +Offset 提交是 **消费者把已处理到的位置记录到 Kafka 内部 Topic 中的机制**。 + +它主要解决: + +1. **消费者重启后从上次位置继续** +2. **消费者组故障恢复** +3. **消费进度可观测** +4. **支持手动回放和重置** + +**自动提交** + +```plain +enable.auto.commit=true +``` + +特点: + +```plain +使用简单 +可能处理前就提交 +异常时可能丢消息 +``` + +**手动提交** + +```plain +处理成功后 commitSync 或 commitAsync +``` + +常见流程: + +```plain +poll records + -> 处理业务 + -> 处理成功 + -> commit offset +``` + +**重复和丢失** + +```plain +先提交,再处理: + 处理失败会丢消息 + +先处理,再提交: + 提交失败会重复消费 +``` + +**需要注意** + ++ Kafka 默认更容易做到至少一次。 ++ 想避免业务重复,必须实现幂等。 ++ offset 提交的是下一条要消费的位置。 ++ 提交粒度过细影响性能,过粗增加重复范围。 + +一句话总结: + +**Offset 提交决定消费者故障恢复位置,可靠消费通常选择处理成功后再提交,并通过业务幂等接受重复消费。** + +# Rebalance +Rebalance 是 **消费者组成员或订阅分区变化时,重新分配 Partition 的过程**。 + +它主要解决: + +1. **消费者扩缩容** +2. **消费者故障接管** +3. **分区新增后的重新分配** +4. **消费负载均衡** + +**触发条件** + +```plain +消费者加入组 +消费者离开组 +消费者心跳超时 +Topic 分区变化 +订阅 Topic 变化 +``` + +**传统流程** + +```plain +暂停消费 + -> 组协调器确认成员 + -> 执行分区分配 + -> 通知消费者新分配 + -> 恢复消费 +``` + +**问题** + +```plain +消费暂停 +重复消费风险 +分区频繁迁移 +消费延迟抖动 +``` + +**优化方向** + +```plain +静态成员 +协作式再均衡 +新版消费者组协议 +合理 session.timeout.ms 和 max.poll.interval.ms +``` + +**需要注意** + ++ Rebalance 不是错误,但频繁 Rebalance 是问题。 ++ 处理时间超过 max.poll.interval.ms 可能被踢出消费者组。 ++ 消费者扩缩容会带来短暂消费抖动。 ++ 需要监控 Rebalance 次数和消费延迟。 + +一句话总结: + +**Rebalance 负责消费者组分区重新分配,是消费弹性和故障接管的基础,但频繁发生会造成延迟和重复消费风险。** + +# 新版 Consumer 协议 +新版 Consumer 协议是 **Kafka 近年来对消费者组协调和 Rebalance 流程的改进方向,目标是降低协调成本、减少客户端复杂度和提升 Rebalance 稳定性**。 + +它主要解决: + +1. **传统 Rebalance 停顿时间较长** +2. **客户端承担过多分配逻辑** +3. **成员变化时影响范围较大** +4. **大规模消费者组协调成本高** + +**传统模型** + +```plain +Consumer Leader 负责分区分配 +Group Coordinator 管理成员 +分配结果再同步给组成员 +``` + +**新版思路** + +```plain +Broker 侧承担更多协调职责 +协议更适合增量变化 +减少全组停顿 +降低客户端分配复杂度 +``` + +**需要注意** + ++ 不同 Kafka 客户端版本对新协议支持程度不同。 ++ 升级前要确认 broker、client、配置兼容性。 ++ 新协议优化 Rebalance,但不消除业务处理慢导致的消费堆积。 ++ 老版本消费者组仍然可能继续使用传统协议。 + +一句话总结: + +**新版 Consumer 协议把消费者组协调能力进一步服务端化,目标是减少大规模消费者组 Rebalance 的停顿和复杂度。** + +# Share Group +Share Group 是 **Kafka 4.x 引入的队列式消费能力方向,也常被称为 Queues for Kafka,用于让多个消费者共享处理同一分区中的不同消息**。 + +它主要解决: + +1. **传统消费者组并行度受分区数限制** +2. **队列场景下不希望过度增加分区** +3. **多个消费者共同处理同一分区消息** +4. **消息级确认和重投递需求** + +**传统 Consumer Group** + +```plain +一个 Partition 同一时刻分配给一个 Consumer +消费者数量超过分区数会空闲 +``` + +**Share Group** + +```plain +多个 Consumer 可以共享同一 Partition 的消息处理 +更接近队列语义 +支持消息级别的处理跟踪 +``` + +**适合场景** + +```plain +任务队列 +异步作业处理 +消费者数量动态变化 +不要求严格分区顺序的工作负载 +``` + +**需要注意** + ++ Share Group 更适合队列式任务,不是替代所有 Consumer Group。 ++ 如果业务强依赖分区内顺序,仍要谨慎使用。 ++ 客户端和 Broker 版本都要确认支持。 ++ 消息级确认也要求业务处理幂等。 + +一句话总结: + +**Share Group 让 Kafka 支持更接近队列的消费模型,突破传统消费者组并行度受分区数限制的问题。** + +# KRaft +KRaft 是 **Kafka Raft Metadata mode,用 Kafka 自身的 Raft 协议管理集群元数据,替代旧 ZooKeeper 架构**。 + +它主要解决: + +1. **去除 ZooKeeper 外部依赖** +2. **统一 Kafka 元数据管理** +3. **提升控制面扩展性和恢复速度** +4. **简化部署和运维** + +**旧架构** + +```plain +Broker + -> ZooKeeper 保存元数据 + -> Controller 监听和管理集群状态 +``` + +**KRaft 架构** + +```plain +Controller Quorum + -> 使用 Raft 管理元数据日志 + -> Broker 从 Controller 获取元数据 +``` + +**角色** + +```plain +broker: + 处理数据读写 + +controller: + 管理元数据、分区状态、Leader 选举 + +broker,controller: + 同一进程同时承担两种角色 +``` + +**需要注意** + ++ Kafka 4.x 新集群应按 KRaft 理解和部署。 ++ Controller Quorum 的节点数和可靠性非常关键。 ++ 元数据日志也需要持久化和备份意识。 ++ 从旧 ZooKeeper 集群迁移要严格遵循版本和迁移流程。 + +一句话总结: + +**KRaft 用 Kafka 自身的 Raft 元数据仲裁替代 ZooKeeper,是 Kafka 4.x 集群控制面的核心架构。** + +# Controller +Controller 是 **Kafka 集群控制面核心角色,负责管理元数据、分区 Leader、Broker 状态和集群变更**。 + +它主要解决: + +1. **分区 Leader 选举** +2. **Broker 上下线处理** +3. **Topic 和 Partition 元数据管理** +4. **副本状态变更** +5. **集群控制事件传播** + +**KRaft Controller Quorum** + +```plain +Controller-1 +Controller-2 +Controller-3 +``` + +通过 Raft 维护元数据日志: + +```plain +metadata log + -> topic 创建 + -> partition leader 变更 + -> broker 注册 + -> 配置变更 +``` + +**Broker 获取元数据** + +```plain +Broker 启动 + -> 注册到 Controller + -> 接收元数据变更 + -> 根据分区角色处理读写和复制 +``` + +**需要注意** + ++ Controller 不直接承载普通消息数据读写,主要负责控制面。 ++ Controller Quorum 不稳定会影响集群管理能力。 ++ Controller 与 Broker 的网络、磁盘、时间配置都要稳定。 ++ 小集群可以 combined 模式,大规模生产更建议角色隔离。 + +一句话总结: + +**Controller 是 Kafka 控制面核心,KRaft 模式下通过 Controller Quorum 和元数据日志管理整个集群状态。** + +# 高水位 HW +高水位 HW 是 **High Watermark,表示消费者可见的最高已提交 Offset 边界**。 + +它主要解决: + +1. **防止消费者读到未复制安全的数据** +2. **定义已提交消息范围** +3. **支持副本故障恢复** +4. **保证消费者读取的一致性** + +**简化示例** + +```plain +Leader log end offset = 10 +Follower-1 log end offset = 10 +Follower-2 log end offset = 8 + +HW = 8 +``` + +消费者只能读到: + +```plain +offset < HW +``` + +**为什么需要 HW** + +如果消费者读到 Leader 上还没复制到足够副本的数据: + +```plain +Leader 宕机 +新 Leader 没有这条消息 +消费者之前读到的数据消失 +``` + +HW 避免这种不一致。 + +**需要注意** + ++ HW 推进依赖副本复制进度。 ++ 副本落后会导致 HW 推进变慢。 ++ 消费延迟不只和消费者有关,也可能和副本同步有关。 ++ 事务场景还会涉及 LSO 等读可见边界。 + +一句话总结: + +**HW 定义消费者可见的已提交日志边界,确保消费者不会读到可能在故障切换中丢失的未安全复制数据。** + +# 顺序性 +Kafka 顺序性是 **Kafka 保证同一 Partition 内消息按 Offset 顺序追加和读取,但不保证 Topic 全局顺序**。 + +它主要解决: + +1. **同一业务实体事件有序** +2. **状态变更按顺序处理** +3. **避免乱序更新导致状态错误** + +**保证同 key 有序** + +```plain +key = orderId + +order_created +order_paid +order_shipped +``` + +通过 key hash 进入同一分区: + +```plain +partition-3: + order_created + order_paid + order_shipped +``` + +**无法天然保证** + +```plain +不同 Partition 之间的全局顺序 +多个 Producer 之间的严格全局顺序 +扩分区后的同 key 历史映射不变 +``` + +**需要注意** + ++ 需要有序的业务必须设计 key。 ++ 消费端并发处理同一分区消息也可能破坏业务顺序。 ++ 重试、异步处理、死信补偿都可能引入乱序。 ++ 全局有序通常意味着单分区,吞吐和可用性会受限。 + +一句话总结: + +**Kafka 只保证分区内顺序,业务要通过合理 key 设计把同一实体事件路由到同一分区。** + +# 可靠性 +Kafka 可靠性是 **消息从生产、存储、复制到消费处理过程中尽量不丢失、可恢复、可重试的能力**。 + +它主要依赖: + +```plain +replication.factor +acks +min.insync.replicas +ISR +幂等生产者 +事务 +手动提交 offset +消费者幂等 +``` + +**高可靠生产** + +```plain +replication.factor=3 +min.insync.replicas=2 +acks=all +enable.idempotence=true +``` + +**高可靠消费** + +```plain +关闭自动提交 +处理成功后提交 offset +业务幂等 +失败重试 +死信或补偿 +``` + +**需要注意** + ++ Kafka 能降低消息丢失概率,但不能替业务处理外部副作用兜底。 ++ 消费端提交 offset 和业务数据库事务之间仍可能不一致。 ++ 高可靠配置会增加延迟并降低部分故障场景下可用性。 ++ 消息至少一次通常比恰好一次更常见,业务幂等必须做。 + +一句话总结: + +**Kafka 可靠性来自副本、ACK、ISR、幂等、事务和正确的 Offset 提交,最终仍需要业务幂等和补偿闭环。** + +# Exactly Once +Exactly Once 是 **Kafka 在特定读写 Kafka 的处理链路中,通过幂等生产者和事务实现的一次性处理语义**。 + +它主要解决: + +1. **生产重试导致重复写入** +2. **消费后生产再提交 Offset 的一致性** +3. **流处理拓扑中重复输出** + +**典型场景** + +```plain +input-topic + -> Kafka Streams 处理 + -> output-topic +``` + +事务保证: + +```plain +输出消息 + + 输入 offset 提交 +作为一个事务提交 +``` + +**需要注意** + ++ Exactly Once 通常限定在 Kafka 到 Kafka 的处理链路内。 ++ 如果写外部数据库、调用 HTTP 接口,仍要外部系统支持幂等或事务。 ++ EOS 会带来额外协调开销。 ++ 不要把 Kafka EOS 理解成所有业务副作用绝对只发生一次。 + +一句话总结: + +**Kafka Exactly Once 主要保证 Kafka 内部消费-处理-生产链路的一致性,外部系统仍需要幂等和事务配合。** + +# Page Cache +Page Cache 是 **Linux 用内存缓存文件数据的机制,也是 Kafka 高吞吐的重要基础**。 + +它主要解决: + +1. **减少磁盘读写延迟** +2. **让顺序写更高效** +3. **利用操作系统缓存热点日志** +4. **降低 JVM 堆内缓存压力** + +**写入流程** + +```plain +Broker 追加消息 + -> 写入 OS Page Cache + -> 后台刷盘 +``` + +**读取流程** + +```plain +Consumer 拉取消息 + -> 如果日志在 Page Cache + -> 直接从内存读取 + -> 否则触发磁盘读取 +``` + +**为什么 Kafka 不把消息都放 JVM 堆** + +```plain +避免 GC 压力 +利用 OS Page Cache +支持大数据量日志缓存 +``` + +**需要注意** + ++ Kafka 写入返回不一定等于立刻 fsync 到磁盘。 ++ Page Cache 被挤压会影响消费读取性能。 ++ Broker 内存规划要给 Page Cache 留空间。 ++ 不要只看 JVM 堆,还要看系统可用内存和磁盘缓存。 + +一句话总结: + +**Kafka 高吞吐大量依赖 Linux Page Cache,合理的内存规划应给操作系统缓存留出足够空间。** + +# 顺序写和零拷贝 +顺序写和零拷贝是 **Kafka 高吞吐的重要底层优化思想**。 + +**顺序写** + +Kafka 追加日志: + +```plain +只在文件末尾追加 +避免大量随机写 +充分利用磁盘顺序写能力 +``` + +**零拷贝** + +消费发送数据时: + +```plain +磁盘文件 + -> Page Cache + -> Socket + -> 网卡 +``` + +减少: + +```plain +内核态到用户态拷贝 +用户态到内核态拷贝 +CPU 拷贝成本 +``` + +**典型机制** + +```plain +sendfile +transferTo +``` + +**需要注意** + ++ 零拷贝不是完全没有拷贝,而是减少不必要的数据拷贝。 ++ 压缩、加密、消息转换可能影响零拷贝路径。 ++ 顺序写不代表磁盘永远不是瓶颈。 ++ 网络带宽常常比磁盘更早成为 Kafka 瓶颈。 + +一句话总结: + +**Kafka 通过追加顺序写和零拷贝减少磁盘随机 IO 与 CPU 拷贝开销,从而获得高吞吐。** + +# Log Retention +Log Retention 是 **Kafka 按时间或大小保留消息日志,超过保留策略后清理旧 Segment 的机制**。 + +它主要解决: + +1. **控制磁盘空间** +2. **支持一段时间内消息回放** +3. **让 Kafka 成为持久事件日志** +4. **自动清理过期数据** + +**常见配置** + +```plain +retention.ms +retention.bytes +segment.ms +segment.bytes +``` + +**清理流程** + +```plain +Segment 达到保留条件 + -> 被标记为可删除 + -> 后台清理线程删除 +``` + +**需要注意** + ++ Retention 是按 Segment 粒度清理,不是逐条消息。 ++ 消费者太慢,消息可能在消费前被清理。 ++ 磁盘容量规划必须结合写入速率和保留时间。 ++ Retention 不等于归档,长期留存可考虑 Tiered Storage。 + +一句话总结: + +**Log Retention 通过时间和大小策略清理旧日志,是 Kafka 控制磁盘空间和支持有限回放窗口的核心机制。** + +# Log Compaction +Log Compaction 是 **Kafka 按 key 保留最新值、清理旧值的日志压缩机制**。 + +它主要解决: + +1. **保存最新状态** +2. **支持状态重建** +3. **减少相同 key 的历史冗余** +4. **适合变更日志和配置类数据** + +**普通日志** + +```plain +key=user1 value=A +key=user1 value=B +key=user1 value=C +``` + +压缩后至少保留: + +```plain +key=user1 value=C +``` + +**删除语义** + +```plain +key=user1 value=null +``` + +称为 tombstone,用于表示删除。 + +**适合场景** + +```plain +用户最新状态 +配置变更 +CDC upsert 事件 +状态存储 changelog +``` + +**需要注意** + ++ Compaction 不保证立刻清理。 ++ 同一个 key 的旧值在一段时间内仍可能存在。 ++ key 不能为空,否则无法按 key 压缩。 ++ 不适合需要完整历史审计的场景。 + +一句话总结: + +**Log Compaction 按 key 保留最新状态,适合状态重建和变更日志,不适合完整历史事件审计。** + +# Tiered Storage +Tiered Storage 是 **Kafka 把较旧日志从本地磁盘分层存储到远端存储的能力,用于降低本地磁盘压力并扩大保留窗口**。 + +它主要解决: + +1. **本地磁盘容量有限** +2. **长时间保留消息成本高** +3. **历史数据回放需求** +4. **冷热数据分层** + +**基本思路** + +```plain +热数据: + 保留在 Broker 本地磁盘 + +冷数据: + 上传到远端对象存储或远端存储系统 +``` + +消费旧数据时: + +```plain +Consumer 请求旧 offset + -> Broker 从远端读取 + -> 返回给 Consumer +``` + +**需要注意** + ++ Tiered Storage 会引入远端存储延迟和成本。 ++ 历史回放性能通常不如本地热数据。 ++ 远端存储可靠性和权限也要纳入运维。 ++ 要区分本地保留策略和远端保留策略。 + +一句话总结: + +**Tiered Storage 通过冷热分层把旧日志放到远端存储,扩大 Kafka 数据保留窗口并降低本地磁盘压力。** + +# Kafka Connect +Kafka Connect 是 **Kafka 的数据集成框架,用于把外部系统数据导入 Kafka,或把 Kafka 数据导出到外部系统**。 + +它主要解决: + +1. **数据库 CDC 接入** +2. **日志和文件采集** +3. **写入搜索、数仓、对象存储** +4. **减少重复开发导入导出程序** + +**Source Connector** + +```plain +外部系统 + -> Connector + -> Kafka Topic +``` + +例如: + +```plain +MySQL CDC -> Kafka +``` + +**Sink Connector** + +```plain +Kafka Topic + -> Connector + -> 外部系统 +``` + +例如: + +```plain +Kafka -> Elasticsearch +Kafka -> S3 +``` + +**需要注意** + ++ Connector 也要考虑 offset、重试、幂等和死信。 ++ Source 和 Sink 的一致性语义取决于外部系统能力。 ++ 大规模 Connect 集群要关注任务分配和 Rebalance。 ++ Connector 配置错误可能导致重复写入或数据延迟。 + +一句话总结: + +**Kafka Connect 用标准化 Connector 把 Kafka 与数据库、搜索、数仓、对象存储等系统连接起来,是数据集成的重要组件。** + +# Kafka Streams +Kafka Streams 是 **Kafka 官方 Java 流处理库,用于基于 Kafka Topic 构建实时处理应用**。 + +它主要解决: + +1. **流式转换** +2. **聚合和窗口** +3. **Join** +4. **状态ful 处理** +5. **Exactly Once 流处理** + +**基本模型** + +```plain +input topic + -> filter/map/groupBy/window/join + -> output topic +``` + +**状态存储** + +```plain +本地 State Store + -> changelog topic + -> 故障后从 changelog 恢复 +``` + +**常见场景** + +```plain +实时统计 +风控规则 +订单状态聚合 +用户行为窗口计算 +流表 Join +``` + +**需要注意** + ++ Kafka Streams 是库,不是独立集群计算框架。 ++ 应用实例数量和 Topic 分区数影响并行度。 ++ 状态存储需要磁盘和恢复时间规划。 ++ 复杂大规模计算可以评估 Flink、Spark Streaming 等框架。 + +一句话总结: + +**Kafka Streams 让应用直接基于 Kafka 构建实时流处理拓扑,适合轻量到中等复杂度的流式计算。** + +# 安全机制 +Kafka 安全机制包括 **认证、授权、加密和审计相关能力**。 + +它主要解决: + +1. **客户端身份确认** +2. **Topic 访问控制** +3. **网络传输加密** +4. **多租户隔离** +5. **操作审计** + +**常见能力** + +```plain +SSL/TLS: + 加密传输和证书认证 + +SASL: + 用户认证机制 + +ACL: + 资源访问控制 + +Principal: + 客户端身份 + +Authorizer: + 授权判断 +``` + +**ACL 示例语义** + +```plain +用户 userA + -> 允许写 Topic order_created + -> 允许读 Group inventory-service +``` + +**需要注意** + ++ 生产环境不要裸奔明文无认证。 ++ ACL 要按最小权限配置。 ++ 证书和密码要有轮换机制。 ++ 多租户 Kafka 要同时做配额、ACL 和 Topic 命名规范。 + +一句话总结: + +**Kafka 安全依赖 TLS、SASL、ACL 和配额等机制,核心目标是确认身份、限制权限和保护数据传输。** + +# 监控指标 +Kafka 监控是 **围绕 Broker、Topic、Partition、Producer、Consumer、Controller 和系统资源建立可观测性的过程**。 + +重点关注: + +```plain +Broker 存活 +Controller 状态 +Under Replicated Partitions +Offline Partitions +ISR 变化 +请求延迟 +生产吞吐 +消费吞吐 +Consumer Lag +磁盘使用率 +网络吞吐 +Page Cache 命中 +GC +``` + +**Consumer Lag** + +```plain +Lag = 最新 offset - 已提交 offset +``` + +表示: + +```plain +消费者落后多少消息 +``` + +**副本健康** + +```plain +Under Replicated Partitions > 0: + 有副本落后或不可用 + +Offline Partitions > 0: + 有分区不可服务 +``` + +**需要注意** + ++ 只看 Broker 是否存活不够。 ++ Lag 高可能是消费者慢,也可能是分区不均、下游慢、Rebalance 频繁。 ++ 磁盘满会直接影响 Broker 稳定性。 ++ Controller 和元数据仲裁健康在 KRaft 模式下非常重要。 + +一句话总结: + +**Kafka 监控要同时关注吞吐、延迟、Lag、副本健康、Controller、磁盘、网络和 JVM,不能只看进程存活。** + +# 常见问题排查 +Kafka 问题通常涉及客户端、Broker、网络、磁盘、Controller、消费者组和业务处理多个层面。 + +**消息堆积** + +排查方向: + +```plain +Consumer Lag 是否持续增长 +消费者实例数是否不足 +分区数是否限制并行度 +业务处理是否变慢 +下游数据库是否慢 +是否频繁 Rebalance +``` + +**生产延迟升高** + +排查方向: + +```plain +acks 配置是否变更 +ISR 是否缩小 +Broker 请求队列是否堆积 +磁盘 IO 是否变慢 +网络是否打满 +batch 和 linger 是否合理 +``` + +**频繁 Rebalance** + +排查方向: + +```plain +消费者是否频繁重启 +poll 间隔是否超过 max.poll.interval.ms +session.timeout.ms 是否太短 +GC 是否过长 +网络是否抖动 +``` + +**副本不同步** + +排查方向: + +```plain +Follower 网络是否慢 +Broker 磁盘 IO 是否慢 +复制线程是否繁忙 +Leader 写入压力是否过高 +跨机房复制是否延迟大 +``` + +**消息重复消费** + +排查方向: + +```plain +是否处理成功但 offset 提交失败 +是否 Rebalance 后重复拉取 +是否生产者重试重复写 +业务是否缺少幂等键 +``` + +**需要注意** + ++ 先区分生产端、Broker 端、消费端、下游业务端。 ++ Kafka 层成功不代表业务处理成功。 ++ 消息重复是常态风险,业务必须幂等。 ++ 排查要看时间线:配置变更、发布、流量峰值、Broker 事件。 + +一句话总结: + +**Kafka 排查要从生产、存储、复制、消费、下游和控制面逐层定位,Lag、ISR、Rebalance、磁盘和网络是最常见线索。** + +# 工程选型 +Kafka 适合高吞吐、可持久化、可回放的事件流场景,但不是所有异步通信都适合 Kafka。 + +**适合 Kafka** + +```plain +日志采集 +埋点数据 +CDC 数据流 +事件驱动架构 +实时计算输入 +多系统订阅同一事件 +高吞吐异步处理 +数据回放 +``` + +**不适合 Kafka** + +```plain +极低延迟 RPC +小规模简单任务队列 +强事务同步调用 +复杂延迟调度 +大量单条消息精细 ACK 的传统队列场景 +``` + +**选型关注** + +```plain +吞吐量 +消息保留时间 +顺序要求 +可靠性要求 +消费并行度 +运维能力 +客户端生态 +监控和告警 +``` + +**需要注意** + ++ Kafka 强项是事件流和日志,不是简单 RPC 替代品。 ++ 分区数、副本数、保留时间要在上线前规划。 ++ 高可靠要接受更高延迟和更复杂配置。 ++ 上线 Kafka 必须配套监控、容量规划和故障演练。 + +一句话总结: + +**Kafka 最适合高吞吐、可持久化、可回放、多订阅的事件流场景,选型时要重点评估顺序、可靠性、延迟、容量和运维成本。** + +# 总结 +Kafka 的核心不是简单消息队列,而是一个围绕分布式日志构建的事件流平台。 + +常见理解路径: + +```plain +基础模型: + Topic、Partition、Replica、Broker + +写入链路: + Producer、Partitioner、Batch、ACK、ISR、幂等、事务 + +存储模型: + Log、Segment、Index、Page Cache、顺序写、零拷贝 + +消费模型: + Consumer、Consumer Group、Offset、Rebalance、Share Group + +控制面: + KRaft、Controller、Metadata Quorum + +可靠性: + 副本、ISR、HW、acks、min.insync.replicas、事务 + +生态: + Kafka Connect、Kafka Streams、Tiered Storage + +运维: + Lag、ISR、Controller、磁盘、网络、GC、Rebalance +``` + +一句话总结: + +**Kafka 的本质是分布式、可复制、可持久化、可回放的日志系统;真正用好 Kafka,要理解分区日志、消费者组、副本同步、KRaft 控制面和端到端可靠性。**