T
- The proto message type.public class DatePartitionedLogger<T extends com.google.protobuf.MessageLite> extends Object
Constructor and Description |
---|
DatePartitionedLogger(com.google.protobuf.Parser<T> parser,
org.apache.hadoop.fs.Path baseDir,
org.apache.hadoop.conf.Configuration conf,
org.apache.hadoop.yarn.util.Clock clock) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.conf.Configuration |
getConfig() |
LocalDate |
getDateFromDir(String dirName)
Extract the date from the directory name, this should be a directory created by this class.
|
String |
getDirForDate(LocalDate date)
Returns the directory name for a given date.
|
String |
getNextDirectory(String currentDir)
Find next available directory, after the given directory.
|
LocalDateTime |
getNow()
Returns the current time, using the underlying clock in UTC time.
|
org.apache.hadoop.fs.Path |
getPathForDate(LocalDate date,
String fileName)
Create a path for the given date and fileName.
|
org.apache.hadoop.fs.Path |
getPathForSubdir(String dirName,
String fileName) |
ProtoMessageReader<T> |
getReader(org.apache.hadoop.fs.Path filePath)
Creates a reader for the given filePath, no validation is done.
|
ProtoMessageWriter<T> |
getWriter(String fileName)
Creates a writer for the given fileName, with date as today.
|
List<org.apache.hadoop.fs.FileStatus> |
scanForChangedFiles(String subDir,
Map<String,Long> currentOffsets)
Returns new or changed files in the given directory.
|
public DatePartitionedLogger(com.google.protobuf.Parser<T> parser, org.apache.hadoop.fs.Path baseDir, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.util.Clock clock) throws IOException
IOException
public ProtoMessageWriter<T> getWriter(String fileName) throws IOException
IOException
public ProtoMessageReader<T> getReader(org.apache.hadoop.fs.Path filePath) throws IOException
IOException
public org.apache.hadoop.fs.Path getPathForDate(LocalDate date, String fileName) throws IOException
IOException
public LocalDate getDateFromDir(String dirName)
public String getDirForDate(LocalDate date)
public String getNextDirectory(String currentDir) throws IOException
IOException
public List<org.apache.hadoop.fs.FileStatus> scanForChangedFiles(String subDir, Map<String,Long> currentOffsets) throws IOException
IOException
public LocalDateTime getNow()
public org.apache.hadoop.conf.Configuration getConfig()
Copyright © 2023 The Apache Software Foundation. All rights reserved.