HangFire循环作业中作业因执行时间太长未完成新作业开启导致重复数据的问题
解决方法:在执行的任务方法前加上Mutex特性即可,如果作业未完成,新作业开启的话,新作业会放入计划中的作业队列中,直到前面的作业完成。
必须使用Hangfire.Pro.Redis 和 Hangfire.SqlServer 作为数据库。
参考:https://github.com/HangfireIO/Hangfire/issues/1053
[Mutex("DownloadVideo")]
public async Task DownloadVideo()
{
}
Mutex特性代码如下:
using System;
using System.Collections.Generic;
using System.Linq;
using Hangfire.Common;
using Hangfire.States;
using Hangfire.Storage; namespace Hangfire.Pro
{
/// <summary>
/// Represents a background job filter that helps to disable concurrent execution
/// without causing worker to wait as in <see cref="Hangfire.DisableConcurrentExecutionAttribute"/>.
/// </summary>
public class MutexAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
{
private static readonly TimeSpan DistributedLockTimeout = TimeSpan.FromMinutes(); private readonly string _resource; public MutexAttribute(string resource)
{
_resource = resource;
RetryInSeconds = ;
} public int RetryInSeconds { get; set; }
public int MaxAttempts { get; set; } public void OnStateElection(ElectStateContext context)
{
// We are intercepting transitions to the Processed state, that is performed by
// a worker just before processing a job. During the state election phase we can
// change the target state to another one, causing a worker not to process the
// backgorund job.
if (context.CandidateState.Name != ProcessingState.StateName ||
context.BackgroundJob.Job == null)
{
return;
} // This filter requires an extended set of storage operations. It's supported
// by all the official storages, and many of the community-based ones.
var storageConnection = context.Connection as JobStorageConnection;
if (storageConnection == null)
{
throw new NotSupportedException("This version of storage doesn't support extended methods. Please try to update to the latest version.");
} string blockedBy; try
{
// Distributed lock is needed here only to prevent a race condition, when another
// worker picks up a background job with the same resource between GET and SET
// operations.
// There will be no race condition, when two or more workers pick up background job
// with the same id, because state transitions are protected with distributed lock
// themselves.
using (AcquireDistributedSetLock(context.Connection, context.BackgroundJob.Job.Args))
{
// Resource set contains a background job id that acquired a mutex for the resource.
// We are getting only one element to see what background job blocked the invocation.
var range = storageConnection.GetRangeFromSet(
GetResourceKey(context.BackgroundJob.Job.Args),
,
); blockedBy = range.Count > ? range[] : null; // We should permit an invocation only when the set is empty, or if current background
// job is already owns a resource. This may happen, when the localTransaction succeeded,
// but outer transaction was failed.
if (blockedBy == null || blockedBy == context.BackgroundJob.Id)
{
// We need to commit the changes inside a distributed lock, otherwise it's
// useless. So we create a local transaction instead of using the
// context.Transaction property.
var localTransaction = context.Connection.CreateWriteTransaction(); // Add the current background job identifier to a resource set. This means
// that resource is owned by the current background job. Identifier will be
// removed only on failed state, or in one of final states (succeeded or
// deleted).
localTransaction.AddToSet(GetResourceKey(context.BackgroundJob.Job.Args), context.BackgroundJob.Id);
localTransaction.Commit(); // Invocation is permitted, and we did all the required things.
return;
}
}
}
catch (DistributedLockTimeoutException)
{
// We weren't able to acquire a distributed lock within a specified window. This may
// be caused by network delays, storage outages or abandoned locks in some storages.
// Since it is required to expire abandoned locks after some time, we can simply
// postpone the invocation.
context.CandidateState = new ScheduledState(TimeSpan.FromSeconds(RetryInSeconds))
{
Reason = "Couldn't acquire a distributed lock for mutex: timeout exceeded"
}; return;
} // Background job execution is blocked. We should change the target state either to
// the Scheduled or to the Deleted one, depending on current retry attempt number.
var currentAttempt = context.GetJobParameter<int>("MutexAttempt") + ;
context.SetJobParameter("MutexAttempt", currentAttempt); context.CandidateState = MaxAttempts == || currentAttempt <= MaxAttempts
? CreateScheduledState(blockedBy, currentAttempt)
: CreateDeletedState(blockedBy);
} public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
if (context.BackgroundJob.Job == null) return; if (context.OldStateName == ProcessingState.StateName)
{
using (AcquireDistributedSetLock(context.Connection, context.BackgroundJob.Job.Args))
{
var localTransaction = context.Connection.CreateWriteTransaction();
localTransaction.RemoveFromSet(GetResourceKey(context.BackgroundJob.Job.Args), context.BackgroundJob.Id); localTransaction.Commit();
}
}
} public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
} private static DeletedState CreateDeletedState(string blockedBy)
{
return new DeletedState
{
Reason = $"Execution was blocked by background job {blockedBy}, all attempts exhausted"
};
} private IState CreateScheduledState(string blockedBy, int currentAttempt)
{
var reason = $"Execution is blocked by background job {blockedBy}, retry attempt: {currentAttempt}"; if (MaxAttempts > )
{
reason += $"/{MaxAttempts}";
} return new ScheduledState(TimeSpan.FromSeconds(RetryInSeconds))
{
Reason = reason
};
} private IDisposable AcquireDistributedSetLock(IStorageConnection connection, IEnumerable<object> args)
{
return connection.AcquireDistributedLock(GetDistributedLockKey(args), DistributedLockTimeout);
} private string GetDistributedLockKey(IEnumerable<object> args)
{
return $"extension:job-mutex:lock:{GetKeyFormat(args, _resource)}";
} private string GetResourceKey(IEnumerable<object> args)
{
return $"extension:job-mutex:set:{GetKeyFormat(args, _resource)}";
} private static string GetKeyFormat(IEnumerable<object> args, string keyFormat)
{
return String.Format(keyFormat, args.ToArray());
}
}
}
HangFire循环作业中作业因执行时间太长未完成新作业开启导致重复数据的问题的更多相关文章
- 解决 ffmpeg 在avformat_find_stream_info执行时间太长
用ffmpeg做demux,网上很多参考文章.对于网络流,avformt_find_stream_info()函数默认需要花费较长的时间进行流格式探测,那么,如何减少探测时间内? 可以通过设置AVFo ...
- SQLServer 删除表中的重复数据
create table Student( ID varchar(10) not null, Name varchar(10) not null, ); insert in ...
- Oracle、SQLServer 删除表中的重复数据,只保留一条记录
原文地址: https://blog.csdn.net/yangwenxue_admin/article/details/51742426 https://www.cnblogs.com/spring ...
- 使用T-SQL找出执行时间过长的作业
有些时候,有些作业遇到问题执行时间过长,因此我写了一个脚本可以根据历史记录,找出执行时间过长的作业,在监控中就可以及时发现这些作业并尽早解决,代码如下: SELECT sj.name , ...
- spark SQL读取ORC文件从Driver启动到开始执行Task(或stage)间隔时间太长(计算Partition时间太长)且产出orc单个文件中stripe个数太多问题解决方案
1.背景: 控制上游文件个数每天7000个,每个文件大小小于256M,50亿条+,orc格式.查看每个文件的stripe个数,500个左右,查询命令:hdfs fsck viewfs://hadoop ...
- 详解C#泛型(二) 获取C#中方法的执行时间及其代码注入 详解C#泛型(一) 详解C#委托和事件(二) 详解C#特性和反射(四) 记一次.net core调用SOAP接口遇到的问题 C# WebRequest.Create 锚点“#”字符问题 根据内容来产生一个二维码
详解C#泛型(二) 一.自定义泛型方法(Generic Method),将类型参数用作参数列表或返回值的类型: void MyFunc<T>() //声明具有一个类型参数的泛型方法 { ...
- 《发际线总是和我作队》第八次团队作业:Alpha冲刺 第五天
项目 内容 这个作业属于哪个课程 软件工程 这个作业的要求在哪里 实验十二 团队作业8:软件测试与Alpha冲刺实验十一 团队作业7:团队项目设计完善&编码 团队名称 发际线总和我作队 作业学 ...
- kettle作业中的js如何写日志文件
在kettle作业中JavaScript脚本有时候也扮演非常重要的角色,此时我们希望有一些日志记录.下面是job中JavaScript记录日志的方式. job的js写日志的方法. 得到日志输出实例 o ...
- 【转】【SQL SERVER】怎样处理作业中的远程服务器错误(42000)
(SQL SERVER)怎样处理作业中的远程服务器错误(42000) 问: 1.我创建了一个链接服务器. 2.在两台服务器之间创建了新的SQL用户. 3.编写了访问链接服务器的SQL语句,执行成功. ...
随机推荐
- 录音--获取语音流(pyAudio)
这是学习时的笔记,包含相关资料链接,有的当时没有细看,记录下来在需要的时候回顾. 有些较混乱的部分,后续会再更新. 欢迎感兴趣的小伙伴一起讨论,跪求大神指点~ 录音-语音流(pyAudio) tags ...
- Docker应用:Hello World
前言: 最近学习了Docker相关技术点,国内关于Docker的资料大多是基于Linux系统的,但是我对Linux又不熟(实际上没用过,掩面哭笑.Jpg). 好在在Win10下也是支持Docker的, ...
- 项目Alpha冲刺(团队6/10)
项目Alpha冲刺(团队6/10) 团队名称: 云打印 作业要求: 项目Alpha冲刺(团队) 作业目标: 完成项目Alpha版本 团队队员 队员学号 队员姓名 个人博客地址 备注 221600412 ...
- 吴恩达机器学习笔记40-用调和平均数F来进行查准率和查全率之间的权衡(Trading Off Precision and Recall by F sore)
在很多应用中,我们希望能够保证查准率和查全率的相对平衡. 我们可以将不同阀值情况下,查全率与查准率的关系绘制成图表,曲线的形状根据数据的不同而不同: 我们希望有一个帮助我们选择这个阀值的方法.一种方法 ...
- 第78节:Java中的网络编程(上)
第78节:Java中的网络编程(上) 前言 网络编程涉及ip,端口,协议,tcp和udp的了解,和对socket通信的网络细节. 网络编程 OSI开放系统互连 网络编程指IO加网络 TCP/IP模型: ...
- 《http权威指南》读书笔记4
概述 最近对http很感兴趣,于是开始看<http权威指南>.别人都说这本书有点老了,而且内容太多.我个人觉得这本书写的太好了,非常长知识,让你知道关于http的很多概念,不仅告诉你怎么做 ...
- 机器学习入门04 - 使用TensorFlow的起始步骤 (First Steps with TensorFlow)
原文链接:https://developers.google.com/machine-learning/crash-course/first-steps-with-tensorflow/ 1- 工具包 ...
- 机器学习入门 - Google的机器学习速成课程
1 - MLCC 通过机器学习,可以有效地解读数据的潜在含义,甚至可以改变思考问题的方式,使用统计信息而非逻辑推理来处理问题. Google的机器学习速成课程(MLCC,machine-learnin ...
- 【翻译】浏览器渲染Rendering那些事:repaint、reflow/relayout、restyle
原文链接:http://www.phpied.com/rendering-repaint-reflowrelayout-restyle/ 有没有被标题中的5个“R”吓到?今天,我们来讨论一下浏览器的渲 ...
- Jenkins系列之七——前端app自动打包
了两周终于搞掂了,期间各种搜教程.各种懵逼,各种坑对小白来还是很难的额.废话不多说直接开撸~~~ 一.介绍下什么是Gradle Gradle是一个基于JVM的构建工具,是一款通用灵活的构建工具,支持m ...