内容纲要
概要描述
本文主要介绍，Hadoop2 的 hdfs 如何开启审计日志，方便判断误删文件之类的操作。
注意: 开启审计日志对 hdfs 性能会有一定影响，默认不会开启，非必要也不建议开启；如果开启，记得及时关闭！
详细说明
- 修改对应版本的 log4j.properties 模版文件
- 配置服务 hdfs ，并重启生效
修改步骤
修改/var/lib/transwarp-manager/master/content/meta/services/HDFS/transwarp-X.Y.Z-final/templates/log4j.properties.raw 文件，将log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit的值，修改为INFO,RFAAUDIT
配置服务，重启hdfs，重启完成之后，确认下/etc/hdfsX/conf/log4j.properties已经修改成功。
执行hdfs语句测试审计功能
1.放置测试文件到/tmp/目录下
[root@argodb1~]$ hadoop fs -put ./ojdbc6.jar /tmp/
2022-07-05 11:07:31,599 INFO util.KerberosUtil: Using principal pattern: HTTP/_HOST
[root@argodb1~]$ hadoop fs -rm -r -f /tmp/ojdbc6.jar
2022-07-05 11:08:20,348 INFO util.KerberosUtil: Using principal pattern: HTTP/_HOST
2022-07-05 11:08:21,561 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
2022-07-05 11:08:21,626 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/tmp/ojdbc6.jar' to trash at: hdfs://nameservice1/user/hdfs/.Trash/Current/tmp/ojdbc6.jar1656990501581
Moved: 'hdfs://nameservice1/tmp/ojdbc6.jar' to trash at: hdfs://nameservice1/user/hdfs/.Trash/Current可以看到，首先会创建_COPYING_文件，再将_COPYING_文件rename成真实的名称。
[root@argodb2/var/log/hdfs1]$ tailf hdfs-audit.log
2022-07-05 11:07:32,947 INFO FSNamesystem.audit: allowed=true ugi=hdfs@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=create src=/tmp/ojdbc6.jar._COPYING_ dst=null perm=hdfs:hadoop:rw-r--r-- proto=rpc
2022-07-05 11:07:34,247 INFO FSNamesystem.audit: allowed=true ugi=hdfs@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=rename src=/tmp/ojdbc6.jar._COPYING_ dst=/tmp/ojdbc6.jar perm=hdfs:hadoop:rw-r--r-- proto=rpc2.将/tmp/ojdbc6.jar文件删除至回收站
[root@argodb1~]$ hadoop fs -rm -r -f /tmp/ojdbc6.jar
2022-07-05 11:08:20,348 INFO util.KerberosUtil: Using principal pattern: HTTP/_HOST
2022-07-05 11:08:21,561 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
2022-07-05 11:08:21,626 INFO fs.TrashPolicyDefault: Moved: 'hdfs://nameservice1/tmp/ojdbc6.jar' to trash at: hdfs://nameservice1/user/hdfs/.Trash/Current/tmp/ojdbc6.jar1656990501581
Moved: 'hdfs://nameservice1/tmp/ojdbc6.jar' to trash at: hdfs://nameservice1/user/hdfs/.Trash/Current可以看到，首先会创建回收站中的对应目录，再将_文件rename到这个目录下加上时间戳。
[root@argodb2/var/log/hdfs1]$ tailf hdfs-audit.log
2022-07-05 11:08:21,575 INFO FSNamesystem.audit: allowed=true ugi=hdfs@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=mkdirs src=/user/hdfs/.Trash/Current/tmp dst=null perm=hdfs:hadoop:rwx------ proto=rpc
2022-07-05 11:08:21,624 INFO FSNamesystem.audit: allowed=true ugi=hdfs@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=rename (options=[TO_TRASH]) src=/tmp/ojdbc6.jar dst=/user/hdfs/.Trash/Current/tmp/ojdbc6.jar1656990501581 perm=hdfs:hadoop:rw-r--r-- proto=rpc执行sql语句测试审计功能
1.创建orc事务表并插入数据
CREATE TABLE EMP_TORC(
EMPNO int,
ENAME string,
JOB string,
MGR INT,
HIREDATE DATE,
SAL INT,
COMM INT,
DEPTNO INT
)CLUSTERED BY (empno) INTO 3 BUCKETS
STORED AS ORC_TRANSACTION;
INSERT INTO EMP_TORC VALUES (7369,'SMITH','CLERK',7902,tdh_todate('17-12-1980','dd-mm-yyyy'),800,NULL,20);
2.执行truncate操作
TRUNCATE TABLE EMP_TORC;2022-07-05 11:35:38,501 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=delete src=/argodbcomputing1/tmp/hive/hive/18d0c7e3-0276-449d-8d23-81ed9cec22e4/hive_2022-07-05_11-35-38_411_9126793528236407643-6 dst=null perm=null proto=rpc
2022-07-05 11:35:38,630 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=getAclStatus src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=null perm=null proto=rpc
2022-07-05 11:35:38,640 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=mkdirs src=/user/hive/.Trash/Current/argodbstorage1/user/hive/warehouse/default.db/hive dst=null perm=hive:hadoop:rwx------ proto=rpc
2022-07-05 11:35:38,648 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=rename (options=[TO_TRASH]) src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=/user/hive/.Trash/Current/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc perm=hive:hive:rwx--x--x proto=rpc
2022-07-05 11:35:38,657 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=mkdirs src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=null perm=hive:hive:rwxr-xr-x proto=rpc
2022-07-05 11:35:38,678 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=setAcl src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=null perm=hive:hive:rwx--x--x proto=rpc
2022-07-05 11:35:38,684 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=getAclStatus src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=null perm=null proto=rpc
3.执行drop table操作
DROP TABLE IF EXISTS EMP_TORC;2022-07-05 11:36:01,726 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=delete src=/argodbcomputing1/tmp/hive/hive/18d0c7e3-0276-449d-8d23-81ed9cec22e4/hive_2022-07-05_11-36-01_648_8496947721525852599-6 dst=null perm=null proto=rpc
2022-07-05 11:36:01,925 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=mkdirs src=/user/hive/.Trash/Current/argodbstorage1/user/hive/warehouse/default.db/hive dst=null perm=hive:hadoop:rwx------ proto=rpc
2022-07-05 11:36:01,935 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=rename (options=[TO_TRASH]) src=/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc dst=/user/hive/.Trash/Current/argodbstorage1/user/hive/warehouse/default.db/hive/emp_torc1656992161924 perm=hive:hive:rwx--x--x proto=rpc
2022-07-05 11:36:02,365 INFO FSNamesystem.audit: allowed=true ugi=hive/argodb1@ARGODBTDH (auth:KERBEROS) ip=/172.22.23.1 cmd=delete src=/argodbcomputing1/tmp/hive/hive/025a7840-8911-4053-96d9-22a7daf0a392/hive_2022-07-05_11-36-02_084_7317455245331088578-9 dst=null perm=null proto=rpc