Package org.apache.nifi.processors.azure.storage

Class PutAzureDataLakeStorage

java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.processor.AbstractProcessor
org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor
org.apache.nifi.processors.azure.storage.PutAzureDataLakeStorage
All Implemented Interfaces:
org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor
@Tags({"azure","microsoft","cloud","storage","adlsgen2","datalake"}) @SeeAlso({DeleteAzureDataLakeStorage.class,FetchAzureDataLakeStorage.class,ListAzureDataLakeStorage.class}) @CapabilityDescription("Writes the contents of a FlowFile as a file on Azure Data Lake Storage Gen 2") @WritesAttribute(attribute="azure.filesystem",description="The name of the Azure File System") @WritesAttribute(attribute="azure.directory",description="The name of the Azure Directory") @WritesAttribute(attribute="azure.filename",description="The name of the Azure File") @WritesAttribute(attribute="azure.primaryUri",description="Primary location for file content") @WritesAttribute(attribute="azure.length",description="The length of the Azure File") @InputRequirement(INPUT_REQUIRED) public class PutAzureDataLakeStorage extends AbstractAzureDataLakeStorageProcessor

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final org.apache.nifi.components.PropertyDescriptor
    BASE_TEMPORARY_PATH
     
    static final org.apache.nifi.components.PropertyDescriptor
    CONFLICT_RESOLUTION
     
    static final String
    FAIL_RESOLUTION
     
    static final String
    IGNORE_RESOLUTION
     
    static long
    MAX_CHUNK_SIZE
     
    private static final List<org.apache.nifi.components.PropertyDescriptor>
    PROPERTIES
     
    static final String
    REPLACE_RESOLUTION
     
    protected static final org.apache.nifi.components.PropertyDescriptor
    WRITING_STRATEGY
     

    Fields inherited from class org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor

    REL_FAILURE, REL_SUCCESS, TEMP_FILE_DIRECTORY

  • Constructor Summary

    Constructors
    Constructor
    Description
    PutAzureDataLakeStorage()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    private Map<String,String>
    createAttributeMap(String fileSystem, String originalDirectory, String fileName, String fileUrl, long length)
     
    private void
    createDirectoryIfNotExists(com.azure.storage.file.datalake.DataLakeDirectoryClient directoryClient)
     
    private com.azure.storage.file.datalake.DataLakeFileClient
    createFile(com.azure.storage.file.datalake.DataLakeDirectoryClient directoryClient, String fileName, String conflictResolution)
    Creates the file on Azure for 'Simple Write' strategy.
    private String
    createPath(String baseDirectory, String path)
     
    private com.azure.storage.file.datalake.DataLakeFileSystemClient
    getFileSystemClient(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.flowfile.FlowFile flowFile, String fileSystem)
     
    protected List<org.apache.nifi.components.PropertyDescriptor>
    getSupportedPropertyDescriptors()
     
    private com.azure.storage.file.datalake.DataLakeFileClient
    handleDataLakeStorageException(com.azure.storage.file.datalake.models.DataLakeStorageException dataLakeStorageException, String destinationPath, String conflictResolution)
     
    void
    onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session)
     
    private void
    removeFile(com.azure.storage.file.datalake.DataLakeFileClient fileClient)
     
    (package private) com.azure.storage.file.datalake.DataLakeFileClient
    renameFile(com.azure.storage.file.datalake.DataLakeFileClient sourceFileClient, String destinationDirectory, String destinationFileName, String conflictResolution)
    This method serves as a "commit" for the upload process in case of 'Write and Rename' strategy.
    private static void
    uploadContent(com.azure.storage.file.datalake.DataLakeFileClient fileClient, InputStream in, long length)
     
    private void
    uploadFile(org.apache.nifi.processor.ProcessSession session, org.apache.nifi.flowfile.FlowFile flowFile, Optional<FileResource> fileResourceFound, long transferSize, com.azure.storage.file.datalake.DataLakeFileClient fileClient)
     

    Methods inherited from class org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor

    getRelationships, getStorageClient, onScheduled, onStopped

    Methods inherited from class org.apache.nifi.processor.AbstractProcessor

    onTrigger

    Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

    Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

    customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

    getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

    Methods inherited from interface org.apache.nifi.processor.Processor

    isStateful, migrateProperties, migrateRelationships

  • Field Details

    • FAIL_RESOLUTION

      public static final String FAIL_RESOLUTION
      See Also:

    • REPLACE_RESOLUTION

      public static final String REPLACE_RESOLUTION
      See Also:

    • IGNORE_RESOLUTION

      public static final String IGNORE_RESOLUTION
      See Also:

    • MAX_CHUNK_SIZE

      public static long MAX_CHUNK_SIZE

    • CONFLICT_RESOLUTION

      public static final org.apache.nifi.components.PropertyDescriptor CONFLICT_RESOLUTION

    • WRITING_STRATEGY

      protected static final org.apache.nifi.components.PropertyDescriptor WRITING_STRATEGY

    • BASE_TEMPORARY_PATH

      public static final org.apache.nifi.components.PropertyDescriptor BASE_TEMPORARY_PATH

    • PROPERTIES

      private static final List<org.apache.nifi.components.PropertyDescriptor> PROPERTIES

  • Constructor Details

    • PutAzureDataLakeStorage

      public PutAzureDataLakeStorage()

  • Method Details

    • getSupportedPropertyDescriptors

      protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()
      Overrides:
      getSupportedPropertyDescriptors in class org.apache.nifi.components.AbstractConfigurableComponent

    • onTrigger

      public void onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session) throws org.apache.nifi.processor.exception.ProcessException
      Specified by:
      onTrigger in class org.apache.nifi.processor.AbstractProcessor
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • getFileSystemClient

      private com.azure.storage.file.datalake.DataLakeFileSystemClient getFileSystemClient(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.flowfile.FlowFile flowFile, String fileSystem)

    • createAttributeMap

      private Map<String,String> createAttributeMap(String fileSystem, String originalDirectory, String fileName, String fileUrl, long length)

    • createDirectoryIfNotExists

      private void createDirectoryIfNotExists(com.azure.storage.file.datalake.DataLakeDirectoryClient directoryClient)

    • uploadFile

      private void uploadFile(org.apache.nifi.processor.ProcessSession session, org.apache.nifi.flowfile.FlowFile flowFile, Optional<FileResource> fileResourceFound, long transferSize, com.azure.storage.file.datalake.DataLakeFileClient fileClient) throws Exception
      Throws:
      Exception

    • uploadContent

      private static void uploadContent(com.azure.storage.file.datalake.DataLakeFileClient fileClient, InputStream in, long length) throws IOException
      Throws:
      IOException

    • createFile

      private com.azure.storage.file.datalake.DataLakeFileClient createFile(com.azure.storage.file.datalake.DataLakeDirectoryClient directoryClient, String fileName, String conflictResolution)
      Creates the file on Azure for 'Simple Write' strategy. Upon upload, a 0-byte file is created, then the payload is appended to it. Because of that, a work-in-progress file is available for readers before the upload is complete.
      Parameters:
      directoryClient - directory client of the uploaded file's parent directory
      fileName - name of the uploaded file
      conflictResolution - conflict resolution strategy
      Returns:
      the file client of the uploaded file or null if the file already exists and conflict resolution strategy is 'ignore'
      Throws:
      org.apache.nifi.processor.exception.ProcessException - if the file already exists and the conflict resolution strategy is 'fail'; also in case of other errors

    • renameFile

      com.azure.storage.file.datalake.DataLakeFileClient renameFile(com.azure.storage.file.datalake.DataLakeFileClient sourceFileClient, String destinationDirectory, String destinationFileName, String conflictResolution)
      This method serves as a "commit" for the upload process in case of 'Write and Rename' strategy. In order to prevent work-in-progress files from being available for readers, a temporary file is written first, and then renamed/moved to its final destination. It is not an efficient approach in case of conflicts because FlowFiles are uploaded unnecessarily, but it is a calculated risk because consistency is more important for 'Write and Rename' strategy.

      Visible for testing

      Parameters:
      sourceFileClient - file client of the temporary file
      destinationDirectory - final location of the uploaded file
      destinationFileName - final name of the uploaded file
      conflictResolution - conflict resolution strategy
      Returns:
      the file client of the uploaded file or null if the file already exists and conflict resolution strategy is 'ignore'
      Throws:
      org.apache.nifi.processor.exception.ProcessException - if the file already exists and the conflict resolution strategy is 'fail'; also in case of other errors

    • handleDataLakeStorageException

      private com.azure.storage.file.datalake.DataLakeFileClient handleDataLakeStorageException(com.azure.storage.file.datalake.models.DataLakeStorageException dataLakeStorageException, String destinationPath, String conflictResolution)

    • createPath

      private String createPath(String baseDirectory, String path)

    • removeFile

      private void removeFile(com.azure.storage.file.datalake.DataLakeFileClient fileClient)