My mfdata module is treating 40000 files per day with classical plugin chain:
[guess_file_type] => [switch] => [plugin_foo] => [switch] => [plugin_bar]
but for a couple of files per day the log of the guess_file_type plugin shows in its log that the tags of the file are removed (in redis) :
2023-02-03T03:41:39.225179Z [INFO] (mfdata.guess_file_type.main#3517739) File /home/mfdata/var/in/tmp/guess_file_type.main/598b888b33a144cca215393e3953d293 moved to /home/mfdata/var/in/step.switch.main/598b888b33a144cca215393e3953d293
2023-02-03T03:41:39.225661Z [INFO] (mfdata.guess_file_type.main#3517739) End of the /home/mfdata/var/in/incoming/PTKG85_HUMI_041200.20230204120000.198491.LT processing after 3 ms
2023-02-03T03:41:39.233049Z [WARNING] (xattrfile#3517739) /home/mfdata/var/in/step.switch.main/598b888b33a144cca215393e3953d293 path does not exist anymore => we removed corresponding attributes in redis
This removal occurs when plugin_foo is treating the file.
So when the file is forwarded to plugin switch, as the tags are lost, the file is forwarded again to plugin_foo... annoying !
I have no idea why the __del__ method of the XattrFile file (https://github.com/metwork-framework/xattrfile/blob/master/xattrfile/__init__.py) is executed after the end of the guess_file_type treatment.
My
mfdatamodule is treating 40000 files per day with classical plugin chain:[guess_file_type]=>[switch]=>[plugin_foo]=>[switch]=>[plugin_bar]but for a couple of files per day the log of the
guess_file_typeplugin shows in its log that the tags of the file are removed (in redis) :This removal occurs when
plugin_foois treating the file.So when the file is forwarded to plugin switch, as the tags are lost, the file is forwarded again to
plugin_foo... annoying !I have no idea why the
__del__method of theXattrFilefile (https://github.com/metwork-framework/xattrfile/blob/master/xattrfile/__init__.py) is executed after the end of the guess_file_type treatment.